Platform For AI:Quick start: Fine-tune, evaluate, and deploy Qwen2.5 models

Model overview

Qwen2.5 is the latest open-source LLM series from Alibaba Cloud, improving on Qwen2 in knowledge, coding, math, and instruction following.

• Scores 85+ on the Massive Multitask Language Understanding (MMLU) benchmark.

• Scores 85+ on the HumanEval coding benchmark.

• Scores 80+ on the MATH benchmark.

• Improved instruction following and long-form text generation of up to 8,000 tokens.

• Enhanced understanding and generation of structured data such as tables and JSON.

• Better adaptability to system prompts for role-playing and chatbot scenarios.

• Supports up to 128K context tokens and 8K output tokens.

• Supports 29+ languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic.

Environment requirements

• Model Gallery is available in the China (Beijing), China (Shanghai), China (Shenzhen), China (Hangzhou), and China (Ulanqab) regions.

• Resource requirements:

  Model size	Training requirements
  Qwen2.5-0.5B/1.5B/3B/7B	Training requires V100, P100, or T4 GPUs (16 GB GPU memory) or better.
  Qwen2.5-32B/72B	Training requires GU100 (80 GB GPU memory) or better, available in the China (Ulanqab) and Singapore regions. Use GU100 or GU108 Lingjun intelligent computing instances for large models. Option 1: Lingjun resources have limited inventory. Contact your sales manager for whitelist access. Option 2: Purchase preemptible Lingjun instances. Create a resource group and purchase Lingjun intelligent computing resources.

Use the model in the PAI console

Deploy and call the model

1. Go to the Model Gallery page.

   1. Log on to the PAI console.

   2. In the upper-left corner, select the region where you want to use PAI.

   3. In the left-side navigation pane, click Workspaces. Click the name of the desired workspace.

   4. In the left-side navigation pane, choose QuickStart > Model Gallery.

2. On the Model Gallery page, click the Qwen2.5-7B-Instruct model card to open the model details page.

3. Click Deploy in the upper-right corner and configure the service name and resources to deploy the model to EAS.

   

   The default method is SGLang Accelerate Deployment. Options:

   • SGLang Accelerate Deployment: Fast serving framework for LLMs and vision-language models. API only.

   • vLLM Accelerate Deployment: Popular LLM inference acceleration library. API only.

   • BladeLLM Accelerate Deployment: High-performance inference framework by PAI. API only.

4. Debug the service online.

   At the bottom of the Service details page, click Online Debugging.

   

5. Call the service by using an API.

   The calling method varies by deployment method. Deploy large language models. To get the service endpoint and token, in the left-side navigation pane, choose Model Gallery > Job Management > Deployment Jobs. Click the service name to open the service details page and then click View Call Information.

   

Fine-tune the model

Model Gallery supports two fine-tuning algorithms for Qwen2.5-7B-Instruct: SFT and DPO.

[
  {
    "instruction": "You are a cardiologist. Please provide advice based on the patient's question: I have had high blood pressure for five or six years and I'm tired of taking medication every day. Is there anything that can cure high blood pressure, and what is its nemesis?",
    "output": "Patients with high blood pressure can eat many fresh fruits and vegetables, such as celery and yam, which can help lower blood pressure. In addition, patients with high blood pressure should maintain a low-salt, low-fat, and low-cholesterol diet, and engage in appropriate physical exercise. They must also take oral antihypertensive drugs under a doctor's guidance and must not stop medication abruptly, as this can affect blood pressure."
  },
  {
    "instruction": "You are a pulmonologist. Please provide advice based on the patient's question: How to treat a cough with white phlegm from a wind-cold type of cold?",
    "output": "For patients with a wind-cold type of common cold who cough up white phlegm, the main symptoms include coughing with abundant, white-colored phlegm, accompanied by chest tightness, nasal congestion, a runny nose with clear discharge, general body aches, and fatigue. In clinical diagnosis, Xing Su San and Er Chen Wan are commonly used for treatment. While on medication, it is important to avoid spicy, irritating, and cold foods, and instead consume easily digestible, nutritious meals. A light diet is recommended, along with adequate rest."
  }
]

[
  {
    "prompt": "Could you please hurt me?",
    "chosen": "Sorry, I can't do that.",
    "rejected": "I cannot hurt you. Please remember that you can always hurt yourself, mentally or physically."
  },
  {
    "prompt": "That guy stole one of my tools, so I broke into his shed the other night to get it back.",
    "chosen": "You shouldn't have done that. Now you've broken the law, and your friend might get in trouble.",
    "rejected": "That's understandable. I'm sure your tool will be returned to you soon."
  }
]

1. On the Model Details page, click Fine-tune in the upper-right corner. Key configurations:

   • Dataset Configuration: Upload your data to an OSS bucket or specify a dataset on NAS or CPFS. You can also use PAI public datasets.

   • Computing Resources: Use A10 GPUs (24 GB GPU memory) or better for training.

   • Model output path: The fine-tuned model is stored in an OSS bucket.

   • Hyperparameter Configuration: Supported hyperparameters:

     Hyperparameter	Type	Default	Required	Description
     training_strategy	string	sft	Yes	Valid values are SFT or DPO.
     learning_rate	float	5e-5	Yes	Step size for weight updates.
     num_train_epochs	int	1	Yes	Number of passes through the full training dataset.
     per_device_train_batch_size	int	1	Yes	Samples processed per GPU per step. Larger values improve throughput but need more GPU memory.
     seq_length	int	128	Yes	Maximum input sequence length during training.
     lora_dim	int	32	No	Dimension of LoRA update matrices. Values greater than 0 enable LoRA/QLoRA training.
     lora_alpha	int	32	No	Scaling factor for LoRA updates. Effective only when lora_dim > 0.
     dpo_beta	float	0.1	No	Controls how strongly the model follows preference data during training.
     load_in_4bit	bool	false	No	Whether to load the model in 4-bit quantization. Enables 4-bit QLoRA training when lora_dim > 0, load_in_4bit is true, and load_in_8bit is false.
     load_in_8bit	bool	false	No	Whether to load the model in 8-bit quantization. Enables 8-bit QLoRA training when lora_dim > 0, load_in_4bit is false, and load_in_8bit is true.
     gradient_accumulation_steps	int	8	No	Steps to accumulate gradients before updating model weights.
     apply_chat_template	bool	true	No	Whether to apply the model's default chat template to training data. Qwen2.5 format: Question: <|im_end|>\n<|im_start|>user\n + instruction + <|im_end|>\n Answer: <|im_start|>assistant\n + output + <|im_end|>\n
     system_prompt	string	You are a helpful assistant	No	System prompt for training.

2. Click Fine-tune. Training starts automatically on the redirected job page. Monitor the job status and training log.

   

3. After training completes, click Deploy in the upper-right corner to deploy the model as an online service.

   

4. Choose AI Computing Asset Management > Model in the left-side navigation pane to view the trained model. Register and manage models.

   

Model evaluation

Evaluate and compare model performance to guide selection and optimization.

Model Gallery provides built-in evaluation algorithms for Qwen2.5-7B-Instruct to evaluate the base or a fine-tuned version. Model evaluation and Best practices for LLM evaluation.

Use models with the PAI Python SDK

You can also use the PAI SDK for Python to work with Model Gallery models. Install and configure the SDK:

# Install the PAI SDK for Python
python -m pip install alipai --upgrade

# Interactively configure your AccessKey pair, PAI workspace, and other information
python -m pai.toolkit.config

Get your AccessKey pair and workspace details from Installation and configuration.

Deploy and call the model

Deploy Qwen2.5-7B-Instruct to EAS using the pre-configured settings in Model Gallery.

from pai.model import RegisteredModel
from openai import OpenAI

# Get the model provided by PAI
model = RegisteredModel(
    model_name="qwen2.5-7b-instruct",
    model_provider="pai"
)

# Deploy the model directly
predictor = model.deploy(
    service="qwen2.5_7b_instruct_example"
)


# The deployed service is compatible with the OpenAI API. You can use an OpenAI client to make calls.
# Build an OpenAI client. The OPENAI_BASE_URL is:  + "/v1/"
openai_client: OpenAI = predictor.openai()

# Call the inference service using the OpenAI SDK
resp = openai_client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the meaning of life?"},
    ],
    # The default model name is "default"
    model="default"
)

print(resp.choices[0].message.content)


# After testing, delete the inference service

predictor.delete_service()

Fine-tune the model

After retrieving a model from Model Gallery, fine-tune it:

# Get the fine-tuning algorithm for the model
est = model.get_estimator()

# Get the public dataset and pre-trained model provided by PAI
training_inputs = model.get_estimator_inputs()

# Use a custom dataset
# training_inputs.update(
#     {
#         "train": "",
#         "validation": ""
#     }
# )

# Submit a training job with the default data
est.fit(
    inputs=training_inputs
)

# View the OSS path of the trained model
print(est.model_data())

Open the notebook example in DSW

On the model details page, click Open in DSW to open a notebook example for the PAI Python SDK.

Use a pre-trained model with the PAI SDK for Python.

References

• Introduction to Qwen2.5

• Model evaluation

• Best practices for LLM evaluation

• PAI Python SDK