Container Service for Kubernetes:Deploy an SGLang inference service with Prefill-Decode Disaggregation

Deploy the inference service with the Prefill-Decode Disaggregation architecture. The figure below shows the interaction sequence diagram for the SGLang Prefill Server and Decode Server.

• When the Prefill Server receives a user's inference request, it creates a Sender object, and the Decode Server creates a Receiver object.

• The Prefill and Decode servers establish a connection through a handshake. The Decode server first allocates a GPU memory address to receive the KV cache. After the Prefill Server completes its computation, it sends the KV cache to the Decode Server. The Decode Server then uses the received KV cache to compute subsequent tokens until the inference request is complete.

Step 2: Deploy the SGLang inference service with Prefill-Decode Disaggregation

This document uses RBG to deploy a 2P1D (2 Prefill, 1 Decode) SGLang inference service. The figure below shows the deployment architecture.

1. Create an sglang_pd.yaml file.

   Sample YAML

   apiVersion: workloads.x-k8s.io/v1alpha1
   kind: RoleBasedGroup
   metadata:
     name: sglang-pd
   spec:
     roles:
       - name: scheduler
         replicas: 1
         dependencies: [ "decode", "prefill" ]
         template:
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
             containers:
               - name: scheduler
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.srt.disaggregation.mini_lb --prefill http://sglang-pd-prefill-0.sglang-pd-prefill:8000 http://sglang-pd-prefill-1.sglang-pd-prefill:8000 --prefill-bootstrap-ports 34000 34000 --decode http://sglang-pd-decode-0.sglang-pd-decode:8000 --host 0.0.0.0 --port 8000
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
   
       - name: prefill
         replicas: 2
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-prefill
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-prefill
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode prefill --port 8000 --disaggregation-bootstrap-port 34000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                   - containerPort: 34000
                     name: bootstrap
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   
       - name: decode
         replicas: 1
         template:
           metadata:
             labels:
               alibabacloud.com/inference-workload: sglang-pd-decode
               alibabacloud.com/inference_backend: sglang
           spec:
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - name: dshm
                 emptyDir:
                   medium: Memory
                   sizeLimit: 15Gi
             containers:
               - name: sglang-decode
                 image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                 imagePullPolicy: Always
                 env:
                   - name: POD_IP
                     valueFrom:
                       fieldRef:
                         fieldPath: status.podIP
                 command:
                   - sh
                   - -c
                   - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode decode --port 8000 --host $(POD_IP) --enable-metrics
                 ports:
                   - containerPort: 8000
                     name: http
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 10
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                   requests:
                     nvidia.com/gpu: "2"
                     aliyun/erdma: 1
                     memory: "16Gi"
                     cpu: "4"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /dev/shm
                     name: dshm
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: sglang-pd
     name: sglang-pd
     namespace: default
   spec:
     ports:
       - name: http
         port: 8000
         protocol: TCP
         targetPort: 8000
     selector:
       rolebasedgroup.workloads.x-k8s.io/name: sglang-pd
       rolebasedgroup.workloads.x-k8s.io/role: scheduler
     type: ClusterIP
   

2. Deploy the SGLang inference service with Prefill-Decode Disaggregation.

   kubectl create -f sglang_pd.yaml

Step 3: Verify the inference service

1. Run the following command to set up port forwarding between the inference service and your local environment.

   Important

   Port forwarding established by kubectl port-forward lacks production-grade reliability, security, and scalability. It is suitable for development and debugging purposes only and should not be used in production environment. For production-ready network solutions in Kubernetes clusters, see Ingress management.

   kubectl port-forward svc/sglang-pd 8000:8000

   Expected output:

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. Run the following command to send a sample inference request to the model inference service.

   curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/models/Qwen3-32B", "messages": [{"role": "user", "content": "test"}], "max_tokens": 30, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   Expected output:

   {"id":"29f3fdac693540bfa7808fc1a8701758","object":"chat.completion","created":1753695366,"model":"/models/Qwen3-32B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user wants me to do a test. I need to first confirm their specific needs. Maybe they want to test my functions, such as answering questions or generating content.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":10,"total_tokens":40,"completion_tokens":30,"prompt_tokens_details":null}}

   This output shows that the model generated a response to the given input.