Deploy standalone LLM inference services - Container Service for Kubernetes

This guide walks you through deploying Qwen3-32B as a standalone large language model (LLM) inference service on Container Service for Kubernetes (ACK). By the end, you'll have a working OpenAI-compatible inference endpoint backed by either vLLM or SGLang running on GPU nodes.

Background

Qwen3-32B

Qwen3-32B represents the latest evolution in the Qwen series, featuring a 32.8B-parameter dense architecture optimized for both reasoning efficiency and conversational fluency.

Key features:

Dual-mode performance: Excels at complex tasks like logical reasoning, math, and code generation, while remaining highly efficient for general text generation.
Advanced capabilities: Demonstrates excellent performance in instruction following, multi-turn dialog, creative writing, and best-in-class tool use for AI agent tasks.
Large context window: Natively handles up to 32,000 tokens of context, which can be extended to 131,000 tokens using YaRN technology.
Multilingual support: Understands and translates over 100 languages, making it ideal for global applications.

For more information, see the blog, GitHub, and documentation.

Prerequisites

Before you begin, ensure that you have:

An ACK cluster running Kubernetes 1.22 or later with GPU-accelerated nodes. See Create an ACK managed cluster and Add GPU nodes to a cluster.

Important

Your GPU nodes must provide more than 64 GB of total GPU memory across the inference pods. The ecs.gn8is-2x.8xlarge instance type (80 GB GPU memory per card) is recommended. For details, see GPU-accelerated compute-optimized instance family gn8is.

Choose an inference engine

This guide supports two inference engines. Choose based on your workload:

Engine	Best for	Key differentiator
vLLM v0.10.0	High-throughput, low-latency serving	PagedAttention KV cache; TP, PP, DP, EP parallelism; OpenAI-compatible API
SGLang 0.4.10.post2	Multimodal and multi-LoRA workloads	RadixAttention prefix cache; PD separation; multi-LoRA batching; FP8/INT4/AWQ/GPTQ quantization

Both engines deploy as a Kubernetes StatefulSet with two GPUs (--tensor-parallel-size 2) and expose a ClusterIP Service on port 8000.

Model deployment

Step 1: Prepare model files

Download the model from ModelScope.

If git-lfs is not installed, run yum install git-lfs or apt-get install git-lfs first. For other installation methods, see Installing Git Large File Storage.

git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Qwen/Qwen3-32B.git
cd Qwen3-32B/
git lfs pull

Upload the model to OSS.

Log in to the OSS console and note your bucket name. If you haven't created one, see Create buckets. Then upload the model files:

For ossutil installation instructions, see Install ossutil.

ossutil mkdir oss://<your-bucket-name>/Qwen3-32B
ossutil cp -r ./Qwen3-32B oss://<your-bucket-name>/Qwen3-32B

Create a persistent volume (PV) and persistent volume claim (PVC).

The PV named llm-model mounts the OSS bucket directory into your inference pods as read-only storage. Choose one of the following methods.

Using the ACK console

Create a PV
- Log in to the ACK console and click Clusters in the left navigation pane.
- On the Clusters page, click the name of your cluster. In the left navigation pane, choose Volumes > Persistent Volumes.
- In the upper-right corner, click Create.
- In the Create PV dialog box, configure the following parameters:
  Parameter
  Value
  PV Type
  OSS
  Volume Name
  llm-model
  Access Certificate
  Your AccessKey ID and AccessKey secret for the OSS bucket
  Bucket ID
  The OSS bucket created in the previous step
  OSS Path
  /Qwen3-32B
Create a PVC
- On the Clusters page, click the name of your cluster. In the left navigation pane, choose Volumes > Persistent Volume Claims.
- In the upper-right corner, click Create.
- In the Create PVC dialog box, configure the following parameters:
  Configuration item
  Value
  PVC Type
  OSS
  Name
  llm-model
  Allocation Mode
  Existing Volumes
  Existing Volumes
  Click Select PV and select the PV you created

Using kubectl

Create a file named llm-model.yaml with the following content. This file defines a Secret with OSS credentials, a static PV pointing to the model directory, and a static PVC that binds to the PV.

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: <your-oss-ak>       # The AccessKey ID used to access the OSS bucket.
  akSecret: <your-oss-sk>   # The AccessKey secret used to access the OSS bucket.
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model
  labels:
    alicloud-pvname: llm-model
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: llm-model
    nodePublishSecretRef:
      name: oss-secret
      namespace: default
    volumeAttributes:
      bucket: <your-bucket-name>       # The bucket name.
      url: <your-bucket-endpoint>      # The endpoint, such as oss-cn-hangzhou-internal.aliyuncs.com.
      otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
      path: <your-model-path>          # The path to the model directory, such as /Qwen3-32B/.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: llm-model
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 30Gi
  selector:
    matchLabels:
      alicloud-pvname: llm-model

Apply the configuration:
```
kubectl create -f llm-model.yaml
```

Step 2: Deploy the inference service

Deploy a standalone inference service with vLLM

Create a file named vllm.yaml:

Expand to view the YAML template

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    # for prometheus to scrape
    alibabacloud.com/inference-workload: vllm-inference
    alibabacloud.com/inference_backend: vllm
  name: vllm-inference
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      alibabacloud.com/inference-workload: vllm-inference
      alibabacloud.com/inference_backend: vllm
  template:
    metadata:
      labels:
        alibabacloud.com/inference-workload: vllm-inference
        alibabacloud.com/inference_backend: vllm
    spec:
      volumes:
      - name: model
        persistentVolumeClaim:
          claimName: llm-model
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 15Gi
      containers:
      - command:
        - sh
        - -c
        - vllm serve /models/Qwen3-32B --port 8000 --trust-remote-code --max-model-len 2048 --gpu-memory-utilization 0.85 --tensor-parallel-size 2
#        image: vllm/vllm-openai:v0.10.0
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0
        name: vllm
        ports:
        - containerPort: 8000
          name: http
        readinessProbe:
          initialDelaySeconds: 30
          periodSeconds: 10
          tcpSocket:
            port: 8000
        resources:
          limits:
            nvidia.com/gpu: "2"
            memory: "16Gi"
            cpu: "4"
          requests:
            nvidia.com/gpu: "2"
            memory: "16Gi"
            cpu: "4"
        volumeMounts:
        - mountPath: /models/Qwen3-32B
          name: model
        - mountPath: /dev/shm
          name: dshm
---
apiVersion: v1
kind: Service
metadata:
  name: inference-service
spec:
  type: ClusterIP
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 8000
  selector:
    alibabacloud.com/inference-workload: vllm-inference

Deploy the service:
```
kubectl create -f vllm.yaml
```

Deploy a standalone inference service with SGLang

Create a file named sglang.yaml:

Expand to view the YAML template

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    # for prometheus to scrape
    alibabacloud.com/inference-workload: sgl-inference
    alibabacloud.com/inference_backend: sglang
  name: sgl-inference
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      alibabacloud.com/inference-workload: sgl-inference
      alibabacloud.com/inference_backend: sglang
  template:
    metadata:
      labels:
        alibabacloud.com/inference-workload: sgl-inference
        alibabacloud.com/inference_backend: sglang
    spec:
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: llm-model
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 15Gi
      containers:
        - command:
            - sh
            - -c
            - "python3 -m sglang.launch_server --model-path /models/Qwen3-32B --tp 2 --trust-remote-code  --context-length 2048 --host 0.0.0.0 --port 8000 --enable-metrics"
          image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
          name: sglang
          ports:
            - containerPort: 8000
              name: http
          readinessProbe:
            initialDelaySeconds: 30
            periodSeconds: 10
            tcpSocket:
              port: 8000
          resources:
            limits:
              nvidia.com/gpu: "2"
              memory: "16Gi"
              cpu: "4"
            requests:
              nvidia.com/gpu: "2"
              memory: "16Gi"
              cpu: "4"
          volumeMounts:
            - mountPath: /models/Qwen3-32B
              name: model
            - mountPath: /dev/shm
              name: dshm
---
apiVersion: v1
kind: Service
metadata:
  name: inference-service
spec:
  type: ClusterIP
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 8000
  selector:
    alibabacloud.com/inference-workload: sgl-inference

Deploy the service:
```
kubectl create -f sglang.yaml
```

Step 3: Verify the inference service

Set up port forwarding to access the service locally:
Important
kubectl port-forward is for development and debugging only. It lacks production-grade reliability, security, and scalability. For production traffic, see Ingress management.
```
kubectl port-forward svc/inference-service 8000:8000
```
Expected output:
```
Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000
```

Send a test request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/models/Qwen3-32B", "messages": [{"role": "user", "content": "Test"}], "max_tokens": 30, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

A successful response looks like:

{"id":"chatcmpl-d490443cd4094bdf86a1a49144f77444","object":"chat.completion","created":1753684011,"model":"/models/Qwen3-32B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"<think>\nOkay, the user sent \"Test\". I need to confirm their request first. They might be testing my functionality or want to see my reaction.","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":40,"completion_tokens":30,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

The model is working correctly when the response contains a choices array with a message.

What's next

Once your inference service is running, consider these next steps for production readiness:

Configure monitoring for LLM inference services: Integrate with Managed Service for Prometheus to collect inference metrics, detect performance bottlenecks, and analyze long-term trends for resource allocation.
Configure auto scaling for LLM inference services: Use the Horizontal Pod Autoscaler (HPA) with ack-alibaba-cloud-metrics-adapter to scale pods automatically based on GPU, CPU, and memory utilization — keeping costs low during idle periods while handling traffic spikes.
Implement Intelligent Routing and Traffic Management with Gateway with Inference Extension: Route requests by model name, manage multiple LoRA variants behind a single endpoint, and assign priority levels to critical models.
Best practices for Fluid data cache optimization policies: Reduce pod cold-start time for large model files (>10 GB) by caching model data across cluster nodes, avoiding repeated downloads from OSS on each restart.

Container Service for Kubernetes:Deploy standalone LLM inference services