Container Service for Kubernetes:Best practice for deploying the DeepSeek full version across multiple nodes in ACK

1.1 Split the model

DeepSeek-R1 provides 671 billion parameters. Each GPU can provide up to 96 GB of memory, which is insufficient to load the entire model. To resolve this issue, you must split the model. In this topic, the TP=8 and PP=2 splitting methods are used. The following figure shows the splitting methods. Model parallelism (PP=2) splits the model into two phases. Each phase runs on a GPU-accelerated node. For example, Model M is split into M1 and M2. M1 runs on the first GPU-accelerated node and passes results to M2 that runs on the second GPU-accelerated node. Data parallelism (TP=8) performs computing operations on eight GPUs in each phase (M1 or M2). In the M1 phase, input data is split into eight portions and processed on eight GPUs. Each GPU processes a portion and then the system merges the computing results from the eight GPUs.

In this topic, vLLM + ray is used to deploy the DeepSeek-R1 model in a distributed manner. The following figure shows the overall deployment architecture. Two vLLM pods are deployed on two ECS instances. Each vLLM pod has eight GPUs. One of the pods functions as a Ray head node and the other pod functions as a Ray worker node.

1.2 Download the model

1.3 Deploy the model

1. Install LeaderWorkerSet.

   1. Log on to the ACK console.

   2. In the navigation pane on the left, click Clusters, then click the name of the cluster you created.

   3. In the navigation pane on the left, click Applications > Helm. On the Helm page, click Deploy.

   4. In the Basic Information step, enter the Application Name and Namespace, find lws in the Chart section, and click Next. In this example, the application name (lws) and namespace (lws-system) are used.

   5. In the Parameters step, select the latest Chart Version, and click OK to install lws.

2. Deploy the model.

   The following figure shows a diagram of the vLLM distributed deployment architecture.

   Deploy models by using Arena

   1. Run the following command to deploy the service:

      arena serve distributed \
              --name=vllm-dist \
              --version=v1 \
              --restful-port=8080 \
              --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0 \
              --readiness-probe-action="tcpSocket" \
              --readiness-probe-action-option="port: 8080" \
              --readiness-probe-option="initialDelaySeconds: 30" \
              --readiness-probe-option="periodSeconds: 30" \
              --share-memory=30Gi \
              --data=llm-model:/models/DeepSeek-R1 \
              --leader-num=1 \
              --leader-gpus=8 \
              --leader-command="bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=\$(LWS_GROUP_SIZE); python3 -m vllm.entrypoints.openai.api_server --model /models/DeepSeek-R1 --port 8080 --trust-remote-code --served-model-name deepseek-r1 --gpu-memory-utilization 0.95 --tensor-parallel-size 8 --pipeline-parallel-size 2 --enforce-eager" \
              --worker-num=1 \
              --worker-gpus=8 \
              --worker-command="bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=\$(LWS_LEADER_ADDRESS)" 

      View the parameter description

      Parameter	Required	Description
      --name	Yes	The name of the inference service. The name must be globally unique.
      --image	Yes	The address of the inference service image.
      --restful-port	Yes	The port of the service.
      --version	No	The version of the service. The default value is the current version.
      --readiness-probe-*	No	Configure readiness detection for a service. The service can be externally provided only after the service is ready.
      --share-memory	No	Configure the shared memory.
      --leader-num	Yes	The number of leader pods. You can set this value only to 1.
      --leader-gpus	No	The number of GPUs used by each leader pod.
      --leader-command	Yes	The command that starts leader pods.
      --data	No	The model address of the service. Format: <pvc-name >:< pod-path>. Run the llm-model:/mnt/models command to mount the llm-model PVC to the /mnt/models directory of the container.

      Expected output:

      configmap/vllm-dist-v1-cm created
      service/vllm-dist-v1 created
      leaderworkerset.leaderworkerset.x-k8s.io/vllm-dist-v1-distributed-serving created
      INFO[0002] The Job vllm-dist has been submitted successfully
      INFO[0002] You can run `arena serve get vllm-dist --type distributed-serving -n default` to check the job status

   2. Run the following command to view the deployment progress of the inference service:

      arena serve get vllm-dist

      Expected output:

      Name:       vllm-dist
      Namespace:  default
      Type:       Distributed
      Version:    v1
      Desired:    1
      Available:  1
      Age:        3m
      Address:    192.168.138.65
      Port:       RESTFUL:8080
      GPU:        16
      
      Instances:
        NAME                                  STATUS   AGE  READY  RESTARTS  GPU  NODE
        ----                                  ------   ---  -----  --------  ---  ----
        vllm-dist-v1-distributed-serving-0    Running  3m   1/1    0         8    cn-beijing.10.x.x.x
        vllm-dist-v1-distributed-serving-0-1  Running  3m   1/1    0         8    cn-beijing.10.x.x.x

   Deploy models by using kubectl

   1. Run the DeepSeek_R1.yaml file to deploy the model service.

      apiVersion: leaderworkerset.x-k8s.io/v1
      kind: LeaderWorkerSet
      metadata:
        name: vllm-dist
      spec:
        replicas: 1
        leaderWorkerTemplate:
          size: 2
          restartPolicy: RecreateGroupOnPodRestart
          leaderTemplate:
            metadata:
              labels: 
                role: leader
            spec:
              volumes:
                - name: model
                  persistentVolumeClaim:
                    claimName: llm-model
                - name: dshm
                  emptyDir:
                    medium: Memory
                    sizeLimit: 15Gi
              containers:
                - name: vllm-leader
                  image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0
                  command:
                    - sh
                    - -c
                    - >-
                      bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE); 
                      python3 -m vllm.entrypoints.openai.api_server --model /models/DeepSeek-R1 --port 8080 --trust-remote-code --served-model-name deepseek-r1 --gpu-memory-utilization 0.95 --tensor-parallel-size 8 --pipeline-parallel-size 2 --enforce-eager
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                    requests:
                      nvidia.com/gpu: "8"
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    initialDelaySeconds: 30
                    periodSeconds: 30
                    tcpSocket:
                      port: 8080
                  volumeMounts:
                    - mountPath: /models/DeepSeek-R1
                      name: model
                    - mountPath: /dev/shm
                      name: dshm
          workerTemplate:
            spec:
              volumes:
                - name: model
                  persistentVolumeClaim:
                    claimName: llm-model
                - name: dshm
                  emptyDir:
                    medium: Memory
                    sizeLimit: 15Gi
              containers:
                - name: vllm-worker
                  image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:v0.10.0
                  command:
                    - sh
                    - -c
                    - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
                  resources:
                    limits:
                      nvidia.com/gpu: "8"
                    requests:
                      nvidia.com/gpu: "8"
                  ports:
                    - containerPort: 8080
                  volumeMounts:
                    - mountPath: /models/DeepSeek-R1
                      name: model
                    - mountPath: /dev/shm
                      name: dshm
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: vllm-dist-v1
      spec:
        type: ClusterIP
        ports:
        - port: 8080
          protocol: TCP
          targetPort: 8080
        selector:
          leaderworkerset.sigs.k8s.io/name: vllm-dist
          role: leader

      kubectl create -f DeepSeek_R1.yaml

   2. Run the following command to view the deployment progress of the inference service:

      kubectl get po |grep vllm-dist

      Expected output:

      NAME            READY   STATUS    RESTARTS   AGE
      vllm-dist-0     1/1     Running   0          20m
      vllm-dist-0-1   1/1     Running   0          20m

3. Create an on-premises port to forward model inference requests.

   1. Run the kubectl port-forward command to configure port forwarding between the on-premises environment and inference service.

      Note

      If you run the kubectl port-forward command to configure port forwarding, the service is not reliable, secure, or extensible in production environments. You can use the service only for development and debugging. Do not run this command to configure port forwarding in production environments. For more information about the networking solutions used to facilitate production in ACK clusters, see Ingress management.

      kubectl port-forward svc/vllm-dist-v1 8080:8080

   2. Send requests to the inference service.

      curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
          "model": "deepseek-r1",
          "prompt": "San Francisco is a",
          "max_tokens": 10,
          "temperature": 0.6
      }'

      Expected output:

      {"id":"cmpl-15977abb0adc44d9aa03628abe9fcc81","object":"text_completion","created":1739346042,"model":"ds","choices":[{"index":0,"text":" city that needs no introduction. Known for its iconic","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":5,"total_tokens":15,"completion_tokens":10,"prompt_tokens_details":null}}

2.1. Configure the DeepSeek model

1. Log on to the Dify platform. Click your profile picture and then click Settings. In the left-side navigation pane, click Model Provider. Find OpenAI-API-compatible and click Add.

2. The following table describes the parameters.

   Parameter	Setting	Note
   Model name	deepseek-r1	You cannot modify this parameter.
   API Key	Example: api-deepseek-r1	You can configure this parameter based on your business requirements.
   API endpoint URL	http://vllm-dist-v1.default:8080/v1	You cannot modify this parameter. The value of this parameter is the name of the local DeepSeek service deployed in the second step.

   

2.2. Create a Q&A assistant

Create a general-purpose AI-powered Q&A assistant. Choose Studio > Create from Blank. Specify a name and a description for the assistant. Use the default settings for other parameters.

2.3 Test the AI-powered Q&A assistant

1. On the right side of the page, you can initiate a conversation with DeepSeek.

   

2. You can integrate the configured DeepSeek Q&A assistant into your personal production environment. For more information, see Apply the AI-powered Q&A assistant to the production environment.