Container Service for Kubernetes:Deploy a vLLM model as an inference service

For more information about vLLM, see vllm-project.

Prerequisites

• An ACK managed cluster or an ACK dedicated cluster with GPU nodes is created. The cluster runs Kubernetes 1.22 or later, and the CUDA version is 12.0 or later. For more information, see Add GPU nodes to a cluster or Create an ACK dedicated cluster with GPU-accelerated nodes.

  (Recommended) Use driver version 525 on GPU nodes. To specify driver version 525.105.17, add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to the GPU node pool. For more information, see Specify a custom GPU driver version for a node.

• The ack-kserve add-on is installed. For more information, see Install the ack-kserve add-on.

• Deploy the cloud-native AI suite is complete.

• The Arena client 0.9.15 or later is installed. For more information, see Configure the Arena client.

• Alibaba Cloud Object Storage Service (OSS) is activated.

Step 1: Prepare the model data and configure a persistent volume

You can prepare the model data in OSS or in NAS. For more information, see Use an ossfs 1.0 static volume and Use static NAS volumes. This tutorial uses OSS.

Step 2: Deploy the inference service

In this step, you check the available GPU resources and then start the vLLM inference service.

1. Run the following command to check the GPU resources that are available in the cluster:

   arena top node

2. Run the following command to start the vllm inference service:

   arena serve kserve \
       --name=qwen \
       --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:0.4.1 \
       --gpus=1 \
       --cpu=4 \
       --memory=12Gi \
       --data="llm-model:/mnt/models/Qwen-7B-Chat-Int8" \
       "python3 -m vllm.entrypoints.openai.api_server --port 8080 --trust-remote-code --served-model-name qwen --model /mnt/models/Qwen-7B-Chat-Int8 --gpu-memory-utilization 0.95 --quantization gptq --max-model-len=6144"

   Parameter	Required	Description
   --name	Yes	The name of the submitted inference service. The name must be globally unique.
   --image	Yes	The image address of the inference service.
   --gpus	No	The number of GPUs that the inference service uses. Default value: 0.
   --cpu	No	The number of CPUs that the inference service uses.
   --memory	No	The amount of memory that the inference service uses.
   --data	No	The model address of the service. In this topic, the model is stored in llm-model, which is mounted to the /mnt/models/ directory in the container.

   Expected output:

   inferenceservice.serving.kserve.io/qwen created
   INFO[0006] The Job qwen has been submitted successfully 
   INFO[0006] You can run `arena serve get qwen --type kserve -n default` to check the job status

Step 3: Verify the inference service

In this step, you confirm the deployment status and send a test request to the inference service.

1. Run the following command to query the deployment status of the KServe inference service:

   arena serve get qwen

   Expected output:

   Click to view the deployment status of the inference service

   Name:       qwen
   Namespace:  default
   Type:       KServe
   Version:    1
   Desired:    1
   Available:  1
   Age:        2m
   Address:    http://qwen-default.example.com
   Port:       :80
   GPU:        1
   
   
   Instances:
     NAME                             STATUS   AGE  READY  RESTARTS  GPU  NODE
     ----                             ------   ---  -----  --------  ---  ----
     qwen-predictor-5485d6d8d5-kvj7g  Running  2m   1/1    0         1    cn-beijing.XX.XX.XX.XX

   The output indicates that the KServe inference is deployed at http://qwen-default.example.com.

2. Run the following command to access the inference service through the Nginx Ingress gateway:

   # Obtain the IP address of the Nginx Ingress.
   NGINX_INGRESS_IP=$(kubectl -n kube-system get svc nginx-ingress-lb -ojsonpath='{.status.loadBalancer.ingress[0].ip}')
   # Obtain the hostname of the inference service.
   SERVICE_HOSTNAME=$(kubectl get inferenceservice qwen -o jsonpath='{.status.url}' | cut -d "/" -f 3)
   # Send a request to access the inference service.
   curl -H "Host: $SERVICE_HOSTNAME" -H "Content-Type: application/json" http://$NGINX_INGRESS_IP:80/v1/chat/completions -d '{"model": "qwen", "messages": [{"role": "user", "content": "Run a test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   Expected output:

   Click to view the expected output

   {"id":"cmpl-b7579597aa284f118718b22b83b726f8","object":"chat.completion","created":1715589652,"model":"qwen","choices":[{"index":0,"message":{"role":"assistant","content":"好的，请问您有什么需要测试的？<|im_end|>"},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":20,"completion_tokens":10}}% 

   The output indicates that the request was sent to the service, and an expected JSON response was returned.

(Optional) Step 4: Delete the inference service

Run the following command to delete the inference service:

arena serve delete qwen

References

• To configure Prometheus monitoring to track the service status and timely resolve issues, see Configure Prometheus monitoring.

• For more information about how to accelerate model loading based on KServe, see Accelerate models using Fluid.