Qwen model inference on ACK using rtp-llm - Container Service for Kubernetes

Deploy a Qwen1.5-4B-Chat inference service on ACK with rtp-llm, using A10 or T4 GPUs.

Background

Qwen1.5-4B-Chat

Qwen1.5-4B-Chat is a 4B-parameter Transformer-based LLM by Alibaba Cloud, trained on web text, domain-specific books, and code. See the Qwen GitHub repository.

rtp-llm

rtp-llm is an LLM inference acceleration engine developed by the Alibaba large model prediction team specifically for LLMs. The engine improves inference efficiency and performance. rtp-llm has the following features:

High-performance CUDA kernels, including PagedAttention, FlashAttention, and FlashDecoding.
WeightOnly INT8 and INT4 quantization.
Support for popular quantization algorithms such as GPTQ (General Purpose Quantization) and AWQ (Approximate Weight Quantization).
Adaptive KVCache quantization framework with detailed optimizations for overhead in dynamic batching.
Optimizations for V100 GPU hardware.

See rtp-llm.

Prerequisites

Before you begin, make sure that you have:

An ACK Pro cluster with GPU-accelerated nodes (Kubernetes 1.22+, ≥16 GB GPU memory per node).
GPU driver version 525 installed on GPU nodes. Add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to pin the driver version.
The latest Arena client is installed.

Step 1: Prepare the model data

Download the Qwen1.5-4B-Chat model, upload it to OSS, and create the PV and PVC in your ACK cluster.

To upload the model to NAS, see Use static NAS persistent volumes.

Download the model files.
1. Run the following command to install Git.
```
# This command is for Red Hat-based systems. For Debian-based systems, use 'apt install git'.
yum install git
```
2. Run the following command to install the Git Large File Storage (LFS) plugin.
```
# This command is for Red Hat-based systems. For Debian-based systems, use 'apt install git-lfs'.
yum install git-lfs
```
3. Run the following command to clone the Qwen1.5-4B-Chat repository from ModelScope to your local machine.
```
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen1.5-4B-Chat.git
```
4. Run the following commands to navigate to the Qwen1.5-4B-Chat directory and pull the large files managed by LFS.
```
cd Qwen1.5-4B-Chat
git lfs pull
```
Upload the downloaded Qwen1.5-4B-Chat files to OSS.
1. Log on to the OSS console, then view and record the name of your bucket.
  
  For information about how to create a bucket, see Create a bucket.
2. Install and configure ossutil to manage OSS resources. For more information, see Install ossutil.
3. Run the following command to create a directory named Qwen1.5-4B-Chat in OSS.
```
ossutil mkdir oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat
```
4. Run the following command to upload the model files to OSS.
```
ossutil cp -r ./Qwen1.5-4B-Chat oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat
```

Configure a PV and a PVC for the destination cluster. For more information, see Use a statically provisioned ossfs 1.0 volume.

The following table describes the sample PV configuration.

Parameter	Description
PV Type	OSS
Name	llm-model
AccessKey	The AccessKey ID and AccessKey Secret used to access OSS.
Bucket ID:	The OSS bucket you created.
OSS path	Select the path where the model is stored, such as /models/Qwen1.5-4B-Chat.

The following table describes the sample PVC configuration.

Parameter

Description

PVC Type

OSS

Name

llm-model

Allocation Mode

Select Existing Volumes.

Existing Volumes

Click Select PV and select the PV that you created.

Step 2: Deploy the inference service

Deploy the inference service for the Qwen1.5-4B-Chat model.

Deploy a custom inference service by using Arena. The service rtp-llm-qwen (v1) runs on one GPU with one replica and a readiness probe. --data mounts the llm-model PVC to /model/Qwen1.5-4B-Chat in the container.

Single A10 GPU

arena serve custom \
    --name=rtp-llm-qwen \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --restful-port=8000 \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/rtp_llm:0.1.12-cuda12-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "MODEL_TYPE=qwen_2 START_PORT=8000 CHECKPOINT_PATH=/model/Qwen1.5-4B-Chat TOKENIZER_PATH=/model/Qwen1.5-4B-Chat python3 -m maga_transformer.start_server"

Single T4 GPU

arena serve custom \
    --name=rtp-llm-qwen \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --restful-port=8000 \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/rtp_llm:0.1.12-cuda12-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "MODEL_TYPE=qwen_2 START_PORT=8000 CHECKPOINT_PATH=/model/Qwen1.5-4B-Chat TOKENIZER_PATH=/model/Qwen1.5-4B-Chat MAX_SEQ_LEN=2048 python3 -m maga_transformer.start_server"

Expected output:

service/rtp-llm-qwen-v1 created
deployment.apps/rtp-llm-qwen-v1-custom-serving created
INFO[0001] The Job rtp-llm-qwen has been submitted successfully
INFO[0001] You can run `arena serve get rtp-llm-qwen --type custom-serving -n default` to check the job status

The inference service is deployed.

View the inference service details and wait for it to become ready.

arena serve get rtp-llm-qwen

Expected output:

Name:       rtp-llm-qwen
Namespace:  default
Type:       Custom
Version:    v1
Desired:    1
Available:  1
Age:        1h
Address:    192.168.XX.XX
Port:       RESTFUL:8000
GPU:        1

Instances:
  NAME                                             STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                             ------   ---  -----  --------  ---  ----
  rtp-llm-qwen-v1-custom-serving-696f699485-mn56v  Running  1h   1/1    0         1    cn-beijing.192.168.XX.XX

The inference service pod rtp-llm-qwen-v1-custom-serving-696f699485-mn56v is running and ready.

Step 3: Verify the inference service

Set up port forwarding to the inference service.
Important
kubectl port-forward is for development and debugging only — not reliable, secure, or scalable for production. For production networking, see Ingress overview.
```
kubectl port-forward svc/rtp-llm-qwen-v1 8000:8000
```
Expected output:
```
Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000
```

Send a request to the inference service.

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/model/Qwen1.5-4B-Chat/", "messages": [{"role": "user", "content": "Run a quick test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"chat-","object":"chat.completion","created":1717383026,"model":"AsyncModel","choices":[{"index":0,"message":{"role":"assistant","content":"Sure. What would you like me to test for you?"},"finish_reason":"stop"}],"usage":{"prompt_tokens":21,"total_tokens":31,"completion_tokens":10}}

The model generated a response to the test message.

(Optional) Step 4: Clean up the environment

If you no longer need the resources, delete them promptly to avoid unnecessary charges.

Delete the model inference service.
```
arena serve del rtp-llm-qwen
```

Delete the PV and PVC.

kubectl delete pvc llm-model
kubectl delete pv llm-model

Appendix: command parameter reference

Parameter	Description	Example
`serve custom`	Arena subcommand. Deploys a custom model service rather than a preset type such as `tfserving` or `triton`.	—
`--name`	Service name. A unique identifier used for subsequent operations such as checking logs and deleting the service.	`modelscope`
`--version`	Service version. A version label for the service, useful for version management and phased releases.	`v1`
`--gpus`	GPU count. The number of GPUs allocated to each pod. Required when the model needs GPUs for inference.	`1`
`--replicas`	Replica count. The number of pods to run. More replicas increase concurrent throughput and availability.	`1`
`--restful-port`	RESTful API port. The port on which the service exposes its RESTful API to receive inference requests.	`8000`
`--readiness-probe-action`	Readiness probe type. The check method used by the Kubernetes readiness probe to determine whether the container is ready to receive traffic.	`tcpSocket`
`--readiness-probe-action-option`	Probe type options. Parameters for the chosen probe type. For `tcpSocket`, specifies the port to check.	`port: 8000`
`--readiness-probe-option`	Additional probe settings. Extra parameters for the readiness probe. This flag can be repeated. Sets the initial delay and check interval.	`initialDelaySeconds: 30`, `periodSeconds: 30`
`--data`	Volume mount. Mounts a PVC at a specified path inside the container, in the format `<pvc-name>:<mount-path>`. Used to mount pre-loaded model files.	`llm-model:/Qwen1.5-4B-Chat`
`--image`	Container image. The full URL of the container image that defines the runtime environment for the service.	`kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/quick-deploy-llm:v1`
`[COMMAND]`	Startup command. The command to run after the container starts. Sets the `MODEL_ID` environment variable and launches `server.py`.	`"MODEL_ID=/Qwen1.5-4B-Chat python3 server.py"`

Parameter	Description
PVC Type	OSS
Name	llm-model
Allocation Mode	Select Existing Volumes.
Existing Volumes	Click Select PV and select the PV that you created.