Container Service for Kubernetes：部署Dynamo PD分離推理服務

部署Dynamo PD分離架構推理服務。Dynamo PD分離時序圖如下所示。

• 使用者請求首先發送給processor組件，經由router選出合適的Decode Worker，將請求轉寄給Decode Worker。

• 由Decode Worker判斷是否prefill計算是在本地完成還是遠程完成。如果需要遠程計算，則向PrefillQueue中發送請求。

• PrefillWorker從Queue擷取請求後進行Prefill計算。等計算完成後，將Prefill KVCache傳輸給Decode Worker。

步驟一：準備Qwen3-32B模型檔案

步驟二：安裝ETCD及NATS服務

Dynamo架構中跨節點通訊使用的是NIXL。NIXL啟動時會向ETCD註冊，從而實現相互發現。NATS服務主要用於Prefill和Decode Worker間訊息傳遞。因此部署推理服務前，需要先部署ETCD及NATS服務。

1. 執行以下命令建立etcd.yaml並部署ETCD服務。

   展開查看相關YAML樣本。

   apiVersion: v1
   kind: Service
   metadata:
     name: etcd
     labels:
       app: etcd
   spec:
     ports:
       - port: 2379
         name: client
       - port: 2380
         name: peer
     clusterIP: None # Enables headless service mode
     selector:
       app: etcd
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: etcd
     labels:
       app: etcd
   spec:
     selector:
       matchLabels:
         app: etcd
     serviceName: "etcd"
     replicas: 1
     template:
       metadata:
         labels:
           app: etcd
       spec:
         containers:
           - name: etcd
             # image: bitnami/etcd:3.5.19
             image: ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/etcd:3.6.1
             volumeMounts:
               - name: data
                 mountPath: /var/lib/etcd
             env:
               - name: ETCDCTL_API
                 value: "3"
               - name: ALLOW_NONE_AUTHENTICATION
                 value: "yes"
         volumes:
           - name: data
             emptyDir: {}

   kubectl apply -f etcd.yaml

2. 執行以下命令建立nats.yaml並部署NATS服務。

   展開查看相關YAML樣本。

   apiVersion: v1
   kind: Service
   metadata:
     name: nats
     labels:
       app: nats
   spec:
     ports:
       - port: 4222
         name: client
       - port: 8222
         name: management
       - port: 6222
         name: cluster
     selector:
       app: nats
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nats
     labels:
       app: nats
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nats
     template:
       metadata:
         labels:
           app: nats
       spec:
         containers:
           - name: nats
             # image: nats:latest
             image: ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/nats:2.11.5
             args:
               - -js
               - --trace
               - -m
               - "8222"
             ports:
               - containerPort: 4222
               - containerPort: 8222
               - containerPort: 6222

   kubectl apply -f nats.yaml

步驟三：部署Dynamo PD分離架構的推理服務

本文使用RBG部署2P1D Dynamo推理服務，部署架構圖如下所示。 Prefill和Decode均採用了TP=2的方式部署。

1. 使用dynamo-configs.yaml配置qwen3模型及Dynamo架構相關的設定檔。

   展開查看相關YAML樣本。

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dynamo-configs
   data:
     pd_disagg.py: |
       from components.frontend import Frontend
       from components.kv_router import Router
       from components.processor import Processor
   
       Frontend.link(Processor).link(Router)
   
     qwen3.yaml: |
       Common:
         model: /models/Qwen3-32B/
         kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
         router: round-robin
         # Number of tokens in a batch for more efficient chunked transfers to GPUs.
         block-size: 128
         max-model-len: 2048
         max-num-batched-tokens: 2048
         disable-log-requests: true
   
       Frontend:
         served_model_name: qwen
         endpoint: dynamo.Processor.chat/completions
         port: 8000
   
       Processor:
         common-configs: [ model, router ]
   
       VllmWorker:
         common-configs: [ model, kv-transfer-config, router, block-size, max-model-len, disable-log-requests ]
         # Enable prefill at different workers.
         remote-prefill: true
         # Disable local prefill so only disaggregated prefill is used.
         conditional-disagg: false
         gpu-memory-utilization: 0.95
         tensor-parallel-size: 1
         ServiceArgs:
           workers: 1
           resources:
             gpu: 1
   
       PrefillWorker:
         common-configs: [ model, kv-transfer-config, block-size, max-model-len, max-num-batched-tokens, gpu-memory-utilization, disable-log-requests ]
         tensor-parallel-size: 1
         gpu-memory-utilization: 0.95
         ServiceArgs:
           workers: 1
           resources:
             gpu: 1
   

   kubectl apply -f dynamo-configs.yaml

2. 準備Dynamo Runtime鏡像。

   具體操作，請參見Dynamo社區文檔擷取，或製作使用vllm作為推理架構的Dynamo Runtime容器鏡像。

3. 執行以下命令建立dynamo.yaml並部署服務。

   展開查看相關YAML樣本。

   apiVersion: workloads.x-k8s.io/v1alpha1
   kind: RoleBasedGroup
   metadata:
     name: dynamo-pd
     namespace: default
   spec:
     roles:
       - name: processor
         replicas: 1
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve graphs.pd_disagg:Frontend -f ./configs/qwen3.yaml
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步驟2中所構建的Dynamo Runtime鏡像地址
                 name: processor
                 ports:
                   - containerPort: 8000
                     name: health
                     protocol: TCP
                   - containerPort: 9345
                     name: request
                     protocol: TCP
                   - containerPort: 443
                     name: api
                     protocol: TCP
                   - containerPort: 9347
                     name: metrics
                     protocol: TCP
                 readinessProbe:
                   initialDelaySeconds: 30
                   periodSeconds: 30
                   tcpSocket:
                     port: 8000
                 resources:
                   limits:
                     cpu: "8"
                     memory: 12Gi
                   requests:
                     cpu: "8"
                     memory: 12Gi
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
                   - mountPath: /workspace/examples/llm/graphs/pd_disagg.py
                     name: dynamo-configs
                     subPath: pd_disagg.py
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
       - name: prefill
         replicas: 2
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve components.prefill_worker:PrefillWorker -f ./configs/qwen3.yaml
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步驟2中所構建的Dynamo Runtime鏡像地址
                 name: prefill-worker
                 resources:
                   limits:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                   requests:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
       - name: decoder
         replicas: 1
         template:
           spec:
             containers:
               - command:
                   - sh
                   - -c
                   - cd /workspace/examples/llm; dynamo serve components.worker:VllmWorker -f ./configs/qwen3.yaml --service-name VllmWorker
                 env:
                   - name: DYNAMO_NAME
                     value: dynamo
                   - name: DYNAMO_NAMESPACE
                     value: default
                   - name: ETCD_ENDPOINTS
                     value: http://etcd:2379
                   - name: NATS_SERVER
                     value: nats://nats:4222
                   - name: DYNAMO_RP_TIMEOUT
                     value: "60"
                 image: #步驟2中所構建的Dynamo Runtime鏡像地址
                 name: vllm-worker
                 resources:
                   limits:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                   requests:
                     cpu: "12"
                     memory: 50Gi
                     nvidia.com/gpu: "2"
                 volumeMounts:
                   - mountPath: /models/Qwen3-32B/
                     name: model
                   - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                     name: dynamo-configs
                     subPath: qwen3.yaml
             volumes:
               - name: model
                 persistentVolumeClaim:
                   claimName: llm-model
               - configMap:
                   name: dynamo-configs
                 name: dynamo-configs
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: dynamo-service
   spec:
     type: ClusterIP
     ports:
       - port: 8000
         protocol: TCP
         targetPort: 8000
     selector:
       rolebasedgroup.workloads.x-k8s.io/name: dynamo-pd
       rolebasedgroup.workloads.x-k8s.io/role: processor

   kubectl apply -f ./dynamo.yaml

步驟四：驗證推理服務

1. 執行以下命令，在推理服務與本地環境之間建立連接埠轉寄。

   重要

   kubectl port-forward建立的連接埠轉寄不具備生產層級的可靠性、安全性和擴充性，因此僅適用於開發和調試目的，不適合在生產環境使用。更多關於Kubernetes叢集內生產可用的網路方案的資訊，請參見Ingress管理。

   kubectl port-forward svc/dynamo-service 8000:8000

   預期輸出：

   Forwarding from 127.0.0.1:8000 -> 8000
   Forwarding from [::1]:8000 -> 8000

2. 執行以下命令，向模型推理服務發送了一條樣本的模型推理請求。

   curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{"model": "qwen","messages": [{"role": "user","content": "測試一下"}],"stream":false,"max_tokens": 30}'

   預期輸出：

   {"id":"31ac3203-c5f9-4b06-a4cd-4435a78d3b35","choices":[{"index":0,"message":{"content":"<think>\n好的，使用者發來“測試一下”，我需要先確認他們的意圖。可能是在測試我的反應速度或者功能，也可能是想","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"length","logprobs":null}],"created":1753702438,"model":"qwen","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":null}

   輸出結果表明模型可以根據給定的輸入（在這個例子中是一條測試訊息）產生相應的回複。