Container Service for Kubernetes:Deploy Qwen model inference on ACK with rtp-llm

Background

Prerequisites

• An ACK Pro cluster with GPU nodes is required. The cluster must run Kubernetes 1.22 or later, and each GPU node must have at least 16 GB of GPU memory. For more information, see Create an ACK managed cluster.

  We recommend that you use NVIDIA driver version 525 on the GPU nodes. You can add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to the GPU node pool to pin the driver version to 525.105.17. For more information, see Customize the GPU driver version of a node by specifying a version number.

• The latest version of the Arena client is installed. For more information, see Configure the Arena client.

Step 1: Prepare the model data

Download the Qwen1.5-4B-Chat model, upload it to OSS, and create the PV and PVC in your ACK cluster.

To upload the model to NAS, see Use static NAS persistent volumes.

1. Download the model files.

   1. Run the following command to install Git.

      # This command is for Red Hat-based systems. For Debian-based systems, use 'apt install git'.
      yum install git

   2. Run the following command to install the Git Large File Storage (LFS) plugin.

      # This command is for Red Hat-based systems. For Debian-based systems, use 'apt install git-lfs'.
      yum install git-lfs

   3. Run the following command to clone the Qwen1.5-4B-Chat repository from ModelScope to your local machine.

      GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen1.5-4B-Chat.git

   4. Run the following commands to navigate to the Qwen1.5-4B-Chat directory and pull the large files managed by LFS.

      cd Qwen1.5-4B-Chat
      git lfs pull

2. Upload the downloaded Qwen1.5-4B-Chat files to OSS.

   1. Log on to the OSS console, then view and record the name of your bucket.

      For information about how to create a bucket, see Create a bucket.

   2. Install and configure ossutil to manage OSS resources. For more information, see Install ossutil.

   3. Run the following command to create a directory named Qwen1.5-4B-Chat in OSS.

      ossutil mkdir oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

   4. Run the following command to upload the model files to OSS.

      ossutil cp -r ./Qwen1.5-4B-Chat oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

3. Configure a PV and a PVC for the destination cluster. For more information, see Use a statically provisioned ossfs 1.0 volume.

   • The following table describes the sample PV configuration.

     Parameter	Description
     PV Type	OSS
     Name	llm-model
     AccessKey	The AccessKey ID and AccessKey Secret used to access OSS.
     Bucket ID:	The OSS bucket you created.
     OSS path	Select the path where the model is stored, such as /models/Qwen1.5-4B-Chat.

   • The following table describes the sample PVC configuration.

     Parameter	Description
     PVC Type	OSS
     Name	llm-model
     Allocation Mode	Select Existing Volumes.
     Existing Volumes	Click Select PV and select the PV that you created.

Step 2: Deploy the inference service

1. Deploy the inference service for the Qwen1.5-4B-Chat model.

   Deploy a custom inference service by using Arena. The service rtp-llm-qwen (v1) runs on one GPU with one replica and a readiness probe. --data mounts the llm-model PVC to /model/Qwen1.5-4B-Chat in the container.

   Single A10 GPU

   arena serve custom \
       --name=rtp-llm-qwen \
       --version=v1 \
       --gpus=1 \
       --replicas=1 \
       --readiness-probe-action="tcpSocket" \
       --readiness-probe-action-option="port: 8000" \
       --readiness-probe-option="initialDelaySeconds: 30" \
       --readiness-probe-option="periodSeconds: 30" \
       --restful-port=8000 \
       --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/rtp_llm:0.1.12-cuda12-ubuntu22.04 \
       --data=llm-model:/model/Qwen1.5-4B-Chat \
       "MODEL_TYPE=qwen_2 START_PORT=8000 CHECKPOINT_PATH=/model/Qwen1.5-4B-Chat TOKENIZER_PATH=/model/Qwen1.5-4B-Chat python3 -m maga_transformer.start_server"

   Single T4 GPU

   arena serve custom \
       --name=rtp-llm-qwen \
       --version=v1 \
       --gpus=1 \
       --replicas=1 \
       --readiness-probe-action="tcpSocket" \
       --readiness-probe-action-option="port: 8000" \
       --readiness-probe-option="initialDelaySeconds: 30" \
       --readiness-probe-option="periodSeconds: 30" \
       --restful-port=8000 \
       --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/rtp_llm:0.1.12-cuda12-ubuntu22.04 \
       --data=llm-model:/model/Qwen1.5-4B-Chat \
       "MODEL_TYPE=qwen_2 START_PORT=8000 CHECKPOINT_PATH=/model/Qwen1.5-4B-Chat TOKENIZER_PATH=/model/Qwen1.5-4B-Chat MAX_SEQ_LEN=2048 python3 -m maga_transformer.start_server"

   Expected output:

   service/rtp-llm-qwen-v1 created
   deployment.apps/rtp-llm-qwen-v1-custom-serving created
   INFO[0001] The Job rtp-llm-qwen has been submitted successfully
   INFO[0001] You can run `arena serve get rtp-llm-qwen --type custom-serving -n default` to check the job status

   The inference service is deployed.

2. View the inference service details and wait for it to become ready.

   arena serve get rtp-llm-qwen

   Expected output:

   Name:       rtp-llm-qwen
   Namespace:  default
   Type:       Custom
   Version:    v1
   Desired:    1
   Available:  1
   Age:        1h
   Address:    192.168.XX.XX
   Port:       RESTFUL:8000
   GPU:        1
   
   Instances:
     NAME                                             STATUS   AGE  READY  RESTARTS  GPU  NODE
     ----                                             ------   ---  -----  --------  ---  ----
     rtp-llm-qwen-v1-custom-serving-696f699485-mn56v  Running  1h   1/1    0         1    cn-beijing.192.168.XX.XX

   The inference service pod rtp-llm-qwen-v1-custom-serving-696f699485-mn56v is running and ready.

Step 3: Verify the inference service

1. Set up port forwarding to the inference service.

   Important

   Port forwarding with kubectl port-forward is not suitable for production because it lacks reliability, security, and scalability. Use it only for development and debugging. For production-grade networking solutions in Kubernetes clusters, see Ingress management.

   kubectl port-forward svc/rtp-llm-qwen-v1 8000:8000

   Expected output:

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. Send a request to the inference service.

   curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/model/Qwen1.5-4B-Chat/", "messages": [{"role": "user", "content": "Run a quick test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   Expected output:

   {"id":"chat-","object":"chat.completion","created":1717383026,"model":"AsyncModel","choices":[{"index":0,"message":{"role":"assistant","content":"Sure. What would you like me to test for you?"},"finish_reason":"stop"}],"usage":{"prompt_tokens":21,"total_tokens":31,"completion_tokens":10}}

   The model generated a response to the test message.

(Optional) Step 4: Clean up the environment

If you no longer need the resources, delete them promptly to avoid unnecessary charges.

• Delete the model inference service.

  arena serve del rtp-llm-qwen

• Delete the PV and PVC.

  kubectl delete pvc llm-model
  kubectl delete pv llm-model

Appendix: command parameter reference