Platform For AI:Train, evaluate, compress, and deploy Qwen2.5-Coder

Model overview

Qwen2.5-Coder is a series of powerful, code-centric models that support a context length of up to 128K tokens and are compatible with 92 programming languages. These models excel at various code-related tasks, including multilingual code generation, code completion, and code repair. The instruction-tuned variant, Qwen2.5-Coder-Instruct, builds on the base model for enhanced performance on these tasks and excellent generalization.

Key features include:

• Multilingual programming capability: Qwen2.5-Coder-Instruct performs exceptionally well on the McEval benchmark, which covers over 40 languages, including niche programming languages.

• Code reasoning capability: The model achieves impressive results on the CRUXEval benchmark, showcasing its strong code reasoning abilities. This enhanced reasoning also correlates with improved performance on complex instruction-following tasks.

• Mathematical capability: As a foundational discipline for coding, mathematics is another area where the model excels, demonstrating its comprehensive capabilities in STEM fields.

• General capabilities: Qwen2.5-Coder-Instruct inherits the powerful general capabilities of the Qwen2.5 base model for stability and broad applicability across a wide range of tasks.

Environment requirements

Deploy and call the model

from openai import OpenAI
# 1. Configure the client.
# Replace <EAS_TOKEN> with the Token from the Invocation Information panel.
openai_api_key = "<EAS_TOKEN>"
# Replace <EAS_ENDPOINT> with the Public Endpoint from the Invocation Information panel.
openai_api_base = "<EAS_ENDPOINT>/v1"
client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
# 2. Get the model name.
# For BladeLLM, set model = "". BladeLLM does not require the model parameter and does not support using client.models.list() to get the model name. Set it to an empty string to comply with the mandatory parameter requirement of the OpenAI SDK.
models = client.models.list()
model = models.data[0].id
print(model)
# 3. Initiate a chat completion request.
# Both streaming (stream=True) and non-streaming (stream=False) outputs are supported.
stream = True
chat_completion = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"},          
    ],
    model=model,
    top_p=0.8,
    temperature=0.7,
    max_tokens=1024,
    stream=stream,
)
if stream:
    for chunk in chat_completion:
        print(chunk.choices[0].delta.content, end="")
else:
    result = chat_completion.choices[0].message.content
    print(result)

Model fine-tuning

Model Gallery provides two out-of-the-box fine-tuning algorithms for the Qwen2.5-Coder-32B-Instruct model: supervised fine-tuning (SFT) and direct preference optimization (DPO).

[
  {
    "instruction": "Create a function to calculate the sum of a sequence of integers.",
    "output": "# Python code\ndef sum_sequence(sequence):\n  sum = 0\n  for num in sequence:\n    sum += num\n  return sum"
  },
  {
    "instruction": "Generate a Python code for crawling a website for a specific type of data.",
    "output": "import requests\nimport re\n\ndef crawl_website_for_phone_numbers(website):\n    response = requests.get(website)\n    phone_numbers = re.findall('\\d{3}-\\d{3}-\\d{4}', response.text)\n    return phone_numbers\n    \nif __name__ == '__main__':\n    print(crawl_website_for_phone_numbers('www.example.com'))"
  }
]

[
  {
    "prompt": "Create a function to calculate the sum of a sequence of integers.",
    "chosen": "# Python code\ndef sum_sequence(sequence):\n  sum = 0\n  for num in sequence:\n    sum += num\n  return sum",
    "rejected": "[x*x for x in [1, 2, 3, 5, 8, 13]]"
  },
  {
    "prompt": "Generate a Python code for crawling a website for a specific type of data.",
    "chosen": "import requests\nimport re\n\ndef crawl_website_for_phone_numbers(website):\n    response = requests.get(website)\n    phone_numbers = re.findall('\\d{3}-\\d{3}-\\d{4}', response.text)\n    return phone_numbers\n    \nif __name__ == '__main__':\n    print(crawl_website_for_phone_numbers('www.example.com'))",
    "rejected": "def remove_duplicates(string): \n    result = \"\" \n    prev = '' \n\n    for char in string:\n        if char != prev: \n            result += char\n            prev = char\n    return result\n\nresult = remove_duplicates(\"AAABBCCCD\")\nprint(result)"
  }
]

1. On the Model Gallery page, click the Qwen2.5-Coder-32B-Instruct model card to go to the model details page.

2. On the model details page, click Train in the upper-right corner. Configure the following parameters:

   • Dataset configuration: After preparing your data, upload it to an Object Storage Service (OSS) bucket or select a dataset on NAS or CPFS by specifying a dataset object. You can also use a public dataset provided by PAI to submit a job and test the algorithm.

   • Computing resource configuration: The algorithm requires a GPU with 80 GB or more VRAM. Ensure that you have sufficient computing resources within your selected resource quota. For the resource specifications required by other model sizes, see Environment requirements.

   • Hyperparameter configuration: The following table lists the training hyperparameters. You can adjust these values or use the defaults.

     Hyperparameter configuration

     Hyperparameter	Type	Default	Required	Description
     training_strategy	String	sft	Yes	The training algorithm. Valid values: SFT and DPO.
     learning_rate	Float	5e-5	Yes	The learning rate, which controls the magnitude of model weight adjustments.
     num_train_epochs	Integer	1	Yes	The number of training epochs.
     per_device_train_batch_size	Integer	1	Yes	The number of samples processed by each GPU per training iteration. A larger batch size can improve efficiency but also increases VRAM requirements.
     seq_length	Integer	128	Yes	The maximum sequence length (in tokens) for model inputs.
     lora_dim	Integer	32	No	The LoRA dimension. If lora_dim > 0, lightweight LoRA or QLoRA training is used.
     lora_alpha	Integer	32	No	The LoRA weight. This parameter is active when lora_dim > 0 (LoRA or QLoRA training is used).
     dpo_beta	Float	0.1	No	Controls the model's reliance on preference information during training.
     load_in_4bit	Boolean	false	No	Specifies whether to load the model in 4-bit precision. If lora_dim > 0, load_in_4bit is set to true, and load_in_8bit is set to false, this enables lightweight 4-bit QLoRA training.
     load_in_8bit	Boolean	false	No	Specifies whether to load the model in 8-bit precision. If lora_dim > 0, load_in_4bit is set to false, and load_in_8bit is set to true, this enables lightweight 8-bit QLoRA training.
     gradient_accumulation_steps	Integer	8	No	The number of steps for gradient accumulation.
     apply_chat_template	Boolean	true	No	Specifies whether to apply the model's default chat template to the training data. For Qwen2 series models, the format is as follows: Question:<|im_end|>\n<|im_start|>user\n + instruction + <|im_end|>\n Answer: <|im_start|>assistant\n + output + <|im_end|>\n
     system_prompt	String	You are a helpful assistant	No	The system prompt used for model training.

3. Click Train. Model Gallery automatically redirects you to the job details page, where you can monitor the training status and view logs.

The trained model is automatically registered in AI Asset - Model Management, where you can view or deploy the model. For more information, see Register and Manage Models.

Model evaluation

PAI provides a built-in evaluation algorithm for the Qwen2.5-Coder-32B-Instruct model. Use this algorithm to assess the performance of the original model and your fine-tuned versions. For more information, see Model Evaluation and Best Practices for Large Language Model Evaluation.

Model compression

After training, you can quantize (compress) the model before deployment to reduce its storage and computing requirements. For more information, see Model Compression.

References

• Qwen2.5-Coder Model Introduction

• Billing for Quick Start