Use vLLM to deploy a Qwen model as an inference service - Container Service for Kubernetes

Prerequisites

Make sure that you have:

An ACK Pro cluster with GPU-accelerated nodes running Kubernetes 1.22 or later, each with at least 16 GB of GPU memory. See Create an ACK managed cluster.
The latest Arena client. See Configure the Arena client.

Note

Use GPU driver version 525. Add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to your GPU-accelerated nodes. See Specify an NVIDIA driver version for nodes by adding a label.

Background

Qwen1.5-4B-Chat

Qwen1.5-4B-Chat is a 4-billion-parameter Transformer-based LLM developed by Alibaba Cloud, trained on web text, domain-specific books, and code. See the Qwen GitHub repository.

vLLM

vLLM is an open-source LLM inference framework optimized for high throughput and low latency. It supports most widely used LLMs, including Qwen, and uses PagedAttention, continuous batching, and quantization to improve efficiency. See the vLLM GitHub repository.

Step 1: Prepare the model data

Download Qwen1.5-4B-Chat, upload it to Object Storage Service (OSS), and create a persistent volume (PV) and persistent volume claim (PVC) in your ACK cluster.

Note

You can also store the model in File Storage NAS instead of OSS. See Mount a statically provisioned NAS volume.

Download the model

Install Git:

   # Run yum install git or apt install git.
   yum install git

Install Git Large File Storage (Git LFS):

   # Run yum install git-lfs or apt install git-lfs.
   yum install git-lfs

Clone Qwen1.5-4B-Chat from ModelScope:

   GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen1.5-4B-Chat.git

Enter the Qwen1.5-4B-Chat directory and pull Git LFS files:
```
   cd Qwen1.5-4B-Chat
   git lfs pull
```

Upload the model to OSS

Log on to the OSS console and note your bucket name. To create one, see Create a bucket.
Install and configure ossutil. See Install ossutil.

Create a Qwen1.5-4B-Chat directory in your OSS bucket:

   ossutil mkdir oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

Upload model files to OSS:

   ossutil cp -r ./Qwen1.5-4B-Chat oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

Create a PV and PVC

Create a PV and PVC in your ACK cluster to mount the model from OSS. See Mount a statically provisioned OSS volume.

PV parameters:

Parameter	Description
PV Type	OSS
Volume Name	llm-model
Access Certificate	AccessKey ID and AccessKey secret for the OSS bucket
Bucket ID	Name of your OSS bucket
OSS Path	Path to the model, such as /models/Qwen1.5-4B-Chat

PVC parameters:

Parameter	Description
PVC Type	OSS
Volume Name	llm-model
Allocation Mode	Select Existing Volumes
Existing Volumes	Click Existing Volumes and select your PV

Step 2: Deploy the inference service

Arena treats model files as a dataset. Use --data to mount the model into the container. In this example, the mount path is /model/Qwen1.5-4B-Chat.

The --max-model-len parameter sets the maximum token length the model can process. Higher values improve interaction quality but consume more GPU memory. See the vLLM code repository on GitHub for all available parameters.

Choose a GPU type

GPU	Use case	max-model-len	Additional parameters
NVIDIA A10	Production (high performance)	16384	None
NVIDIA T4	Testing (lower cost)	8192	`--dtype half`

Deploy the service

Choose a command based on your GPU type.

NVIDIA A10 (recommended for production)

arena serve custom \
    --name=vllm-qwen-4b-chat \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/vllm:0.4.1-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "python3 -m vllm.entrypoints.openai.api_server --trust-remote-code --model /model/Qwen1.5-4B-Chat/ --gpu-memory-utilization 0.95 --max-model-len 16384"

NVIDIA T4 (for testing)

arena serve custom \
    --name=vllm-qwen-4b-chat \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/vllm:0.4.1-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "python3 -m vllm.entrypoints.openai.api_server --trust-remote-code --model /model/Qwen1.5-4B-Chat/ --gpu-memory-utilization 0.95 --max-model-len 8192 --dtype half"

Note

T4 uses --max-model-len 8192 (vs. 16384 for A10) and adds --dtype half to fit T4 memory constraints.

Arena parameters:

Parameter	Description
`--name`	Inference service name
`--version`	Inference service version
`--gpus`	GPUs per replica
`--replicas`	Number of replicas
`--restful-port`	Port for the inference service
`--readiness-probe-action`	Readiness probe type. Valid values: `HttpGet`, `Exec`, `gRPC`, and `TCPSocket`
`--readiness-probe-action-option`	Readiness probe connection method
`--readiness-probe-option`	Readiness probe configuration
`--data`	Mounts a shared PVC. Format: `PVCName:MountPath`. Use `arena data list` to query available PVCs
`--image`	Container image for the inference service

Expected output:

service/vllm-qwen-4b-chat-v1 created
deployment.apps/vllm-qwen-4b-chat-v1-custom-serving created
INFO[0008] The Job vllm-qwen-4b-chat has been submitted successfully
INFO[0008] You can run `arena serve get vllm-qwen-4b-chat --type custom-serving -n default` to check the job status

Verify the deployment

Check the inference service status:

arena serve get vllm-qwen-4b-chat

Expected output:

Name:       vllm-qwen-4b-chat
Namespace:  default
Type:       Custom
Version:    v1
Desired:    1
Available:  1
Age:        36m
Address:    172.16.XX.XX
Port:       RESTFUL:8000
GPU:        1

Instances:
  NAME                                                  STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                                  ------   ---  -----  --------  ---  ----
  vllm-qwen-4b-chat-v1-custom-serving-6d7c786b9f-z6nfk  Running  36m  1/1    0         1    cn-beijing.192.168.XX.XX

When Available matches Desired and the instance status is Running, the service is ready.

Step 3: Test the inference service

Set up port forwarding

Important

Port forwarding with kubectl port-forward is for development and debugging only. For production, use an Ingress.

Forward port 8000 to your local machine:

kubectl port-forward svc/vllm-qwen-4b-chat-v1 8000:8000

Expected output:

Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000

Send a test request

Open a new terminal and send an inference request:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/model/Qwen1.5-4B-Chat/", "messages": [{"role": "user", "content": "Test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"cmpl-503270b21fa44db2b6b3c3e0abaa3c02","object":"chat.completion","created":1717141209,"model":"/model/Qwen1.5-4B-Chat/","choices":[{"index":0,"message":{"role":"assistant","content":"OK. What do you want to test?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":21,"total_tokens":30,"completion_tokens":9}}

vLLM exposes an OpenAI-compatible API at the /v1/chat/completions endpoint.

(Optional) Clean up resources

Delete unused resources to avoid unnecessary costs.

Delete the inference service:

arena serve delete vllm-qwen-4b-chat

Delete the PVC and PV:

kubectl delete pvc llm-model
kubectl delete pv llm-model

Next steps

Configure an Ingress to expose the inference service in production.
To deploy other Qwen models, adjust --max-model-len and --gpu-memory-utilization based on model size and GPU memory.
See the vLLM GitHub repository for configuration options.