Container Service for Kubernetes:Deploy an SGLang inference service with Prefill-Decode Disaggregation

When to use this architecture

Prefill/Decode separation assigns the two core stages of LLM inference to dedicated GPUs. Evaluate whether your workload benefits from this approach before committing to the additional infrastructure.

Choose Prefill/Decode separation when:

• Your workload has long input prompts and requires consistent, low inter-token latency.

• You need to tune time-to-first-token (TTFT) and time per output token (TPOT) independently.

• Tail latency spikes during decoding are unacceptable.

Choose standard (aggregated) inference when:

• Your input prompts are short and output sequences are long.

• Latency consistency is less critical than maximizing raw throughput.

• You have fewer than 6 GPUs with 32 GB memory each.

How it works

LLM inference has two stages with different compute profiles:

• Prefill stage: Processes the entire user prompt in one pass. The model calculates attention for all input tokens in parallel and generates the initial KV cache. This stage is compute-intensive and runs once per request.

• Decode stage: Generates tokens one at a time based on the KV cache. Each step loads large model weights and the KV cache from GPU memory, making this stage memory-bound.

When both stages run on the same GPU, they compete for resources. Compute-heavy prefill tasks increase latency for decode tasks in the same batch, leading to higher TPOT and reduced efficiency.

Prefill/Decode separation solves this by running each stage on dedicated GPUs:

The following sequence shows how the prefill and decode servers interact for each request:

1. A user request arrives at the scheduler, which routes it to a prefill server.

2. The prefill server processes the full prompt, generates the KV cache, and creates a Sender object.

3. The decode server creates a Receiver object and allocates GPU memory for the incoming KV cache.

4. The prefill and decode servers complete a handshake. The prefill server transfers the KV cache to the decode server over RDMA.

5. The decode server generates tokens autoregressively until the response is complete.

This separation lets each GPU pool optimize for its stage's compute profile, reducing TPOT and increasing system throughput.

Prerequisites

Before you begin, ensure that you have:

• An ACK managed cluster (version 1.22 or later) with GPU nodes. For more information, see Create an ACK managed cluster and Add GPU nodes to a cluster

• At least 6 GPUs with at least 32 GB of memory each. The framework relies on GPUDirect RDMA for KV cache transfer between prefill and decode servers

• Node specifications that support elastic RDMA (eRDMA). Use the ecs.ebmgn8is.32xlarge ECS Bare Metal Instance type. For more information, see ECS Bare Metal Instance types

• Alibaba Cloud Linux 3 64-bit OS image (pre-installed with the eRDMA software stack). For more information, see Add eRDMA nodes in ACK

• The ACK eRDMA Controller component installed and configured. For more information, see Use eRDMA to accelerate container networks

• The ack-rbgs component installed (see Install ack-rbgs below)

Step 1: Prepare the Qwen3-32B model files

Step 2: Deploy the inference service

This deployment uses RBG to create a 2P1D architecture: 2 prefill server replicas, 1 decode server replica, and 1 scheduler. Each server uses tensor parallelism across 2 GPUs, totaling 6 GPUs.

The following table explains the key parameters in the deployment:

1. Create a file named sglang_pd.yaml:

   Sample YAML

      apiVersion: workloads.x-k8s.io/v1alpha1
      kind: RoleBasedGroup
      metadata:
        name: sglang-pd
      spec:
        roles:
          # --- Scheduler role ---
          # Routes incoming requests to prefill servers.
          # Depends on both decode and prefill roles being ready.
          - name: scheduler
            replicas: 1
            dependencies: [ "decode", "prefill" ]
            template:
              spec:
                volumes:
                  - name: model
                    persistentVolumeClaim:
                      claimName: llm-model
                containers:
                  - name: scheduler
                    image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                    command:
                      - sh
                      - -c
                      - python3 -m sglang.srt.disaggregation.mini_lb --prefill http://sglang-pd-prefill-0.sglang-pd-prefill:8000 http://sglang-pd-prefill-1.sglang-pd-prefill:8000 --prefill-bootstrap-ports 34000 34000 --decode http://sglang-pd-decode-0.sglang-pd-decode:8000 --host 0.0.0.0 --port 8000
                    volumeMounts:
                      - mountPath: /models/Qwen3-32B/
                        name: model
   
          # --- Prefill role ---
          # Processes input prompts and generates the KV cache.
          # 2 replicas, each using 2 GPUs with tensor parallelism.
          - name: prefill
            replicas: 2
            template:
              metadata:
                labels:
                  alibabacloud.com/inference-workload: sglang-pd-prefill
                  alibabacloud.com/inference_backend: sglang
              spec:
                volumes:
                  - name: model
                    persistentVolumeClaim:
                      claimName: llm-model
                  - name: dshm
                    emptyDir:
                      medium: Memory
                      sizeLimit: 15Gi
                containers:
                  - name: sglang-prefill
                    image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                    imagePullPolicy: Always
                    env:
                      - name: POD_IP
                        valueFrom:
                          fieldRef:
                            fieldPath: status.podIP
                    command:
                      - sh
                      - -c
                      - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode prefill --port 8000 --disaggregation-bootstrap-port 34000 --host $(POD_IP) --enable-metrics
                    ports:
                      - containerPort: 8000
                        name: http
                      - containerPort: 34000
                        name: bootstrap
                    readinessProbe:
                      initialDelaySeconds: 30
                      periodSeconds: 10
                      tcpSocket:
                        port: 8000
                    resources:
                      limits:
                        nvidia.com/gpu: "2"
                        aliyun/erdma: 1
                        memory: "16Gi"
                        cpu: "4"
                      requests:
                        nvidia.com/gpu: "2"
                        aliyun/erdma: 1
                        memory: "16Gi"
                        cpu: "4"
                    volumeMounts:
                      - mountPath: /models/Qwen3-32B/
                        name: model
                      - mountPath: /dev/shm
                        name: dshm
   
          # --- Decode role ---
          # Generates output tokens autoregressively from the KV cache.
          # 1 replica using 2 GPUs with tensor parallelism.
          - name: decode
            replicas: 1
            template:
              metadata:
                labels:
                  alibabacloud.com/inference-workload: sglang-pd-decode
                  alibabacloud.com/inference_backend: sglang
              spec:
                volumes:
                  - name: model
                    persistentVolumeClaim:
                      claimName: llm-model
                  - name: dshm
                    emptyDir:
                      medium: Memory
                      sizeLimit: 15Gi
                containers:
                  - name: sglang-decode
                    image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/anolis-docker-images/docker-temp:0.3.4.post2-sglang0.4.10.post2-pytorch2.7.1.8-cuda12.8.1-py312-alinux3.2104
                    imagePullPolicy: Always
                    env:
                      - name: POD_IP
                        valueFrom:
                          fieldRef:
                            fieldPath: status.podIP
                    command:
                      - sh
                      - -c
                      - python3 -m sglang.launch_server --tp 2 --model-path /models/Qwen3-32B/ --disaggregation-mode decode --port 8000 --host $(POD_IP) --enable-metrics
                    ports:
                      - containerPort: 8000
                        name: http
                    readinessProbe:
                      initialDelaySeconds: 30
                      periodSeconds: 10
                      tcpSocket:
                        port: 8000
                    resources:
                      limits:
                        nvidia.com/gpu: "2"
                        aliyun/erdma: 1
                        memory: "16Gi"
                        cpu: "4"
                      requests:
                        nvidia.com/gpu: "2"
                        aliyun/erdma: 1
                        memory: "16Gi"
                        cpu: "4"
                    volumeMounts:
                      - mountPath: /models/Qwen3-32B/
                        name: model
                      - mountPath: /dev/shm
                        name: dshm
      ---
      apiVersion: v1
      kind: Service
      metadata:
        labels:
          app: sglang-pd
        name: sglang-pd
        namespace: default
      spec:
        ports:
          - name: http
            port: 8000
            protocol: TCP
            targetPort: 8000
        selector:
          rolebasedgroup.workloads.x-k8s.io/name: sglang-pd
          rolebasedgroup.workloads.x-k8s.io/role: scheduler
        type: ClusterIP

2. Deploy the service:

      kubectl create -f sglang_pd.yaml

3. Wait for all Pods to reach the Ready state:

      kubectl get pods -l rolebasedgroup.workloads.x-k8s.io/name=sglang-pd --watch

Step 3: Verify the inference service

kubectl port-forward svc/sglang-pd 8000:8000

Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/Qwen3-32B",
    "messages": [{"role": "user", "content": "test"}],
    "max_tokens": 30,
    "temperature": 0.7,
    "top_p": 0.9,
    "seed": 10
  }'

{
  "id": "29f3fdac693540bfa7808fc1a8701758",
  "object": "chat.completion",
  "created": 1753695366,
  "model": "/models/Qwen3-32B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<think>\nOkay, the user wants me to do a test. I need to first confirm their specific needs. Maybe they want to test my functions, such as answering questions or generating content.",
        "reasoning_content": null,
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "matched_stop": null
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "total_tokens": 40,
    "completion_tokens": 30,
    "prompt_tokens_details": null
  }
}

References

• Configure Prometheus monitoring for LLM inference services Integrate with Managed Service for Prometheus to collect inference metrics, detect failures and performance bottlenecks, troubleshoot issues with real-time data, and analyze long-term performance trends for resource optimization.

• Accelerate model loading with Fluid distributed caching Large model files (>10 GB) stored in OSS or File Storage NAS can cause slow Pod startups due to long download times. Fluid creates a distributed caching layer across cluster nodes to accelerate model loading through parallel data throughput and reduced I/O latency from local caching.

• Implement intelligent routing and traffic management by using Gateway with Inference Extension ACK Gateway with Inference Extension provides model-aware load balancing, intelligent model routing based on request payload, and request prioritization for production inference workloads.