Platform For AI:PAI-ChatLearn: Best practices for Qwen3 reinforcement learning

1.1 Prepare the development environment

Before you begin, make sure that you have completed the following tasks:

1. Activate PAI and create a default workspace.

2. Purchase Lingjun resources and create a resource quota. This example requires Lingjun resources. This example requires two machines with the ml.gx8xf.8xlarge-gu108 node specification. For more information about the node specifications for Lingjun resources, see Billing of AI computing resources.

3. Create a dataset to store the required training and result files.

   • Dataset Type: Select Basic.

   • Storage Type: Select a file storage type. This topic uses File Storage (General-purpose NAS). If you do not have a NAS file system, see Create a file system.

     Note

     If your training task requires high read and write speeds and performance, use File Storage (Intelligent Computing CPFS).

   • Default Mount Path: Use the default value /mnt/data/.

4. Create a Data Science Workshop (DSW) instance with the following key parameters.

   • Resource Type: Select Resource Quota.

   • Resource Quota: Select the resource quota that you created for the Lingjun resources.

   • Instance Type: Configure the following resource specifications.

     • GPUs: 8 or more.

     • vCPUs: 90.

     • Memory (GiB): 1024.

     • Shared Memory (GiB): 1024.

   • Image Information: Select Image Address and set the runtime image to dsw-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/pai-training-algorithm/chatlearn:torch2.5.1-vllm0.6.6-ubuntu22.04-cuda12.6-py310. You must change the region in the registry address to the region where your DSW instance is located. For example, if you are using a DSW instance in Shanghai, change the region in the address to cn-shanghai.

   • Dataset Mounting: Click Custom Dataset, select the dataset that you created, and use the default mount path.

5. If you use a Resource Access Management (RAM) user to perform the following operations, you must grant the RAM user the required permissions to use DSW, Deep Learning Containers (DLC), or Elastic Algorithm Service (EAS). For more information, see Product dependencies and authorization: DSW, Product dependencies and authorization: DLC, or Product dependencies and authorization: EAS.

1.2 Download the code repository

1. Go to the PAI-DSW development environment.

   1. Log on to the PAI console. In the upper-left corner of the page, select a region. China (Ulanqab) is recommended.

   2. In the navigation pane on the left, click Workspace List. On the page that appears, click the name of the workspace that you want to manage.

   3. In the navigation pane on the left, select Model Training > Data Science Workshop (DSW), and click Open in the Actions column of the target instance.

2. In the top menu bar, click Terminal. On the tab that appears, click Create Terminal.

3. Download the ChatLearn code repository.

   git clone https://github.com/alibaba/ChatLearn.git && cd ChatLearn && git checkout 4ad5912306df5d4a814dc2dd5567fcb26f5d473b

1.3 Prepare the Qwen3 model

Download the Qwen3 model weights from ModelScope.

modelscope download --model Qwen/Qwen3-8B --local_dir Qwen3-8B

1.4 Prepare the training dataset

This example uses the MATH-lighteval dataset to demonstrate the ChatLearn reinforcement learning workflow.

• This dataset is for math reasoning tasks. It uses fixed rules for reward score validation.

• To perform reinforcement learning on a custom task, you can implement a custom reward scoring function by referencing examples/fsdp/models/rule_reward.py in the ChatLearn code repository.

# Download the dataset
mkdir -p dataset
modelscope download --dataset AI-ModelScope/MATH-lighteval --local_dir dataset/MATH-lighteval
# Pre-process the dataset
python examples/fsdp/data/data_preprocess/math_lighteval.py --input_dir dataset/MATH-lighteval --local_dir dataset/MATH-lighteval

2.1 Single-node training in DSW

Continue in the DSW environment and run the following command to start training. The trained model is saved to the mounted dataset for later deployment.

bash examples/fsdp/scripts/train_grpo_qwen3.sh

2.2 Multi-node training in DLC

After you develop and test on a single node, you can configure a distributed task on multi-GPU servers in the DLC environment to speed up model training.

1. Go to the Create Task page.

   1. Log on to the PAI console. At the top of the page, select the destination region. On the right, select the destination workspace, and then click Enter DLC.

   2. On the Deep Learning Containers (DLC) page, click Create Task.

2. On the Create Job page, configure the following key parameters. Use the default values for other parameters. For more information, see Create a training task.

   Parameter		Description
   Basic Information	Job Name	A custom task name. This example uses test_qwen3_dlc.
   Environment Information	Image Information	Select Image Address and enter the following address in the text box: dsw-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/pai-training-algorithm/chatlearn:torch2.5.1-vllm0.6.6-ubuntu22.04-cuda12.6-py310. You must change the region in the registry address to match your current region.
   	Mount dataset	Click Custom Dataset, select the dataset that you created, and use the default mount path /mnt/data/.
   	Startup Command	Configure the following command. The startup parameters for the train_grpo_qwen3.sh script are the same as those for single-node pre-training in DSW. cd /mnt/data/ChatLearn && bash examples/fsdp/scripts/train_grpo_qwen3.sh
   Resource Information	Resource Type	Select Lingjun Intelligent Computing.
   	Source	Select Resource Quota.
   	Resource Quota	Select the resource quota that you created for the Lingjun resources.
   	Framework	Select PyTorch.
   	Job Resource	On the Worker node configuration tab, configure the following parameters: Quantity: 2. For multi-node training, set Number of Nodes to the number of machines that you need. GPUs: 8 vCPUs: 90 Memory (GiB): 1024 Shared Memory (GiB): 1024

3. Click OK. You are redirected to the Deep Learning Containers (DLC) page. You can click the task name to view the task execution status on the Task Details page. When the Status changes to Succeeded, the training task is complete.

   Note

   If the DLC job fails and the error message ray.exceptions.RpcError: Timed out while waiting for GCS to become available. appears, the training task has actually completed. You can proceed to deploy the saved model as a service.

2.3 Main parameter descriptions

3.1 Deploy the model service

1. Log on to the PAI console. At the top of the page, select the destination region. On the right, select the destination workspace, and then click Enter EAS.

2. Click Deploy Service. In the Custom Model Deployment area, click Custom Deployment.

3. On the Custom Deployment page, configure the following key parameters. Use the default values for other parameters.

   Parameter		Description
   Basic Information	Service Name	A custom name for the model service. The name must be unique within the same region. This example uses test_qwen3.
   Environment Information	Deployment Method	Select Image-based Deployment.
   	Image Configuration	Select Image Address and enter the following registry address in the text box: eas-registry-vpc.cn-wulanchabu.cr.aliyuncs.com/pai-eas/vllm:v0.8.5.post1. You must change the region in the registry address to match your current region.
   	Mount storage	Select General-purpose NAS and configure the following parameters: Select File System: Select the NAS file system that you used to create the dataset. File System Mount Target: Select the mount target that you used to create the dataset. File System Path: The path to the Hugging Face model stored in NAS. This example uses /ChatLearn/output/qwen3-grpo/save_model/policy_trainer/20/huggingface/. Mount Path: The path after mounting. This example uses /qwen3_rlhf.
   	Command	Set to vllm serve /qwen3_rlhf --host 0.0.0.0 --port 8000 --max-model-len 8192. Note If you deploy on a V100 instance, set the run command to vllm serve /qwen3_rlhf --host 0.0.0.0 --port 8000 --max-model-len 8192 --dtype=half.
   	Port Number	Set to 8000.
   Resource Information	Resource Type	Select Public Resources.
   	Number of Replicas	Configure this based on the model and the selected resources. For an 8B model, set this to 1.
   	Deployment	For the resource specification, select an A10 or V100 instance. This example uses ecs.gn7i-c32g1.8xlarge.
   Network Information	VPC	After you configure the NAS mount target, the system automatically matches the VPC and vSwitch with the preset NAS file system. Set the security group as needed.
   	vSwitch
   	Security group

4. Click Deploy. The service deployment takes about 6 minutes. When the Service Status changes to Running, the service deployment is complete.

3.2 Call the service

1. Obtain the service endpoint and token. On the Inference Service tab, find the target service and navigate to the Overview page. In the Basic Information section, click View Endpoint Information.

2. Use the following code to call the service. Replace <YOUR EAS URL> with the endpoint that you obtained in the previous step. Set the token as an environment variable.

   import os
   from openai import OpenAI
   
   # Set the token as an environment variable.
   openai_api_key = os.environ.get("Token")
   # Replace <YOUR EAS URL> with the service endpoint.
   openai_api_base = "<YOUR EAS URL>/v1"
   
   client = OpenAI(
       api_key=openai_api_key,
       base_url=openai_api_base,
   )
   
   chat_response = client.chat.completions.create(
       model="/qwen3_rlhf",
       messages=[
           {"role": "system", "content": "You are a helpful assistant."},
           {"role": "user", "content": "Find the smallest positive integer solution to $\\tan{19x^{\\circ}}=\\dfrac{\\cos{96^{\\circ}}+\\sin{96^{\\circ}}}{\\cos{96^{\\circ}}-\\sin{96^{\\circ}}}$. Let's think step by step and output the final answer within \\boxed{}."},
       ],
       temperature=0.7,
       top_p=0.8,
       presence_penalty=1.5,
       extra_body={
           "top_k": 20, 
           "chat_template_kwargs": {"enable_thinking": False},
       }
   )
   print("Chat response:", chat_response)