Container Service for Kubernetes:Deploy an SGLang inference service with Prefill/Decode separation

Deploy the inference service with the Prefill/Decode separation architecture. The following time series chart shows the interaction between the SGLang Prefill Server and Decode Server.

• When a user inference request is received, the Prefill Server creates a Sender object, and the Decode Server creates a Receiver object.

• The Prefill and Decode servers establish a connection through a handshake. The Decode Server first allocates a GPU memory address to receive the KV cache. After the Prefill Server completes its computation, it sends the KV cache to the Decode Server. The Decode Server then continues to compute subsequent tokens until the user's inference request is complete.

Step 1: Prepare the Qwen3-32B model files

Step 2: Deploy the SGLang inference service with Prefill/Decode separation

This topic uses RBG to deploy a 2P1D SGLang inference service. The deployment architecture is shown in the following figure.

1. Create an sglang_pd.yaml file.

   Expand to view the sample YAML code.

   apiVersion: workloads.x-k8s.io/v1alpha1
   kind: RoleBasedGroup
   metadata:
     name: sglang-pd
   spec:
     roles:
       - name: scheduler
         replicas: 1
         dependencies: [ "decode", "prefill" ]
         template:
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
             containers:
               - name: scheduler
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.srt.disaggregation.mini_lb --prefill http://sglang-pd-prefill-0.sglang-pd-prefill:8000 http://sglang-pd-prefill-1.sglang-pd-prefill:8000 --prefill-bootstrap-ports 34000 34000 --decode http://sglang-pd-decode-0.sglang-pd-decode:8000 --host 0.0.0.0 --port 8000
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
   
       - name: prefill
         replicas: 2
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-prefill
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-prefill
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode prefill --port 8000 --disaggregation-bootstrap-port 34000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                   - containerPort: 34000
                     name: bootstrap
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   
       - name: decode
         replicas: 1
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-decode
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-decode
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode decode --port 8000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: sglang-pd
     name: sglang-pd
     namespace: default
   spec:
     ports:
       - name: http
         port: 8000
         protocol: TCP
         targetPort: 8000
     selector:
       rolebasedgroup.workloads.x-k8s.io/name: sglang-pd
       rolebasedgroup.workloads.x-k8s.io/role: scheduler
     type: ClusterIP
   

2. Deploy the SGLang inference service with Prefill/Decode separation.

   kubectl create -f sglang_pd.yaml

Step 3: Verify the inference service

1. Run the following command to establish port forwarding between the inference service and your local environment.

   Important

   Port forwarding established by kubectl port-forward lacks production-grade reliability, security, and scalability. It is suitable for development and debugging purposes only and should not be used in a production environment. For production-ready network solutions in Kubernetes clusters, see Ingress management.

   kubectl port-forward svc/sglang-pd 8000:8000

   Expected output:

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. You can run the following command to send a sample inference request to the model inference service.

   curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/models/Qwen3-32B", "messages": [{"role": "user", "content": "test"}], "max_tokens": 30, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   Expected output:

   {"id":"29f3fdac693540bfa7808fc1a8701758","object":"chat.completion","created":1753695366,"model":"/models/Qwen3-32B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user wants me to do a test. I need to first confirm their specific needs. Maybe they want to test my functions, such as answering questions or generating content.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":10,"total_tokens":40,"completion_tokens":30,"prompt_tokens_details":null}}

   The output indicates that the model can generate a response based on the input test message.