Container Service for Kubernetes:Securely Deploy vLLM Inference Services in ACK Heterogeneous Confidential Computing Clusters

You can use ACK-CAI to deploy vLLM model inference services in ACK heterogeneous confidential computing clusters. This ensures secure isolation and encryption protection for models and data. The advantages are as follows:

• Hardware-level security isolation: Build a hardware-level trusted execution environment (TEE) using Intel® TDX and NVIDIA GPU TEE technology. This ensures the confidentiality and integrity of models and data during computation.

• Trusted key distribution: Strictly verify the runtime environment using a remote attestation mechanism. After successful verification, the dedicated Trustee service distributes model decryption keys to the trusted environment.

• End-to-end data encryption: Establish a client-to-server encrypted channel using a Trusted Network Gateway (TNG). This protects the security of inference requests and response data during transmission.

• Non-intrusive to applications: Automatically inject security components into Pods based on Kubernetes Webhook. Simply enable security capabilities for applications using Annotation, without modifying business code or images.

How it works

ACK-CAI provides transparent confidential computing capabilities for application Pods by dynamically injecting a set of Sidecar containers named Trustiflux. Its core security mechanism relies on remote attestation, ensuring that models and data are accessed only in a trusted environment.

Process and environment guide

Deploying and accessing a secure vLLM inference service involves the following stages:

Step 1: Prepare encrypted models

This section uses an encryption tool to process model data and uploads it to Object Storage Service (OSS), preparing for subsequent encrypted distribution.

Execution environment: To achieve secure isolation, prepare a temporary ECS instance to download, encrypt, and upload models. We recommend that the ECS instance is in the same region as the OSS Bucket for high-speed upload of encrypted model data over the private network.

Model files are large, and the process takes a long time. To quickly experience the solution, skip this section. You can use the encrypted model example files for a trial, and proceed directly to Step 2: Deploy the Trustee remote attestation service.

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

Step 2: Deploy the Trustee remote attestation service

Following the zero trust principle, any confidential computing environment must pass verification before gaining permissions to access sensitive data, such as model decryption keys. This step deploys a dedicated Trustee service to verify the runtime environment of models and inference services. This ensures that model decryption keys are injected only when the environment is confirmed trustworthy, and verifies the environment's trustworthiness when the client-side initiates an inference request.

Execution environment: A dedicated, standalone server deployed outside the ACK cluster, such as an ECS instance or an on-premises private server.

Step 3: Configure the ACK confidential computing cluster

This section builds an underlying infrastructure with hardware-level security isolation capabilities for running confidential computing tasks. It involves creating an ACK cluster and adding ecs.gn8v-tee instances with Intel TDX and NVIDIA TEE capabilities as worker nodes.

Execution environment: ECS, ACK console (for creating clusters, node pools, and ECS instances), and the Shell environment of the created ecs.gn8v-tee instance (for installing drivers).

1. Create an ACK managed cluster Pro edition in the China (Beijing) region. For more information, see Create an ACK managed cluster.

2. Create a node pool for the cluster to manage confidential computing instances. For more information, see Create and manage node pools.

   • vSwitch: Select the virtual switch in China (Beijing) Zone L.

   • Scaling Mode: Keep default configurations. Do not enable automatic elastic scaling.

   • Instance type: ecs.gn8v-tee.4xlarge and above.

   • Operating System: Alibaba Cloud Linux 3.2104 LTS 64-bit.

   • System Disk: 100 GiB or more.

   • Expected Number of Nodes: The initial number of nodes in the node pool. Keep default configurations, which is 0.

   • Node Labels: Add labels (Key: ack.aliyun.com/nvidia-driver-version, Value: 550.144.03) to specify the NVIDIA driver version.

3. Create EGS confidential computing instances as cluster nodes. For more information, see Custom purchase instances.

   • Region: China (Beijing).

   • Network and zone: VPC consistent with the cluster VPC, Zone L.

   • Instance type: ecs.gn8v-tee.4xlarge and above.

     gn8v-tee instance types have CPU and GPU confidential computing features enabled by default. No need to additionally select confidential virtual machines.

   • Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.

4. Log on to the created EGS instance and install NVIDIA drivers and CUDA toolkit. For more information, see Step 1: Install NVIDIA drivers and CUDA toolkit.

5. Add the EGS instance to the previously created node pool. Select manual addition as the method. For more information, see Add existing nodes.

Step 4: Deploy ACK-CAI components

Based on ACK-CAI components, enable non-intrusive confidential computing capabilities for applications in the cluster. This component includes a Webhook controller that automatically injects Sidecar containers into Pods based on their Annotation. These Sidecar containers provide remote attestation, model decryption, and secure communication.

Execution environment: ACK console.

1. Log on to the Container Service Management Console . In the navigation pane on the left, click Clusters.

2. On the Clusters page, click the name of your cluster. In the navigation pane on the left, click Applications > Helm.

3. Click Create and follow the on-screen prompts to install the latest version of ACK-CAI.

   After installation is complete, view the deployment status in the Helm Chart list.

Step 5: Deploy the vLLM model inference service

After the basic environment and security components are ready, this section uses Helm to deploy the vLLM service. Add specific Annotations to declare that the application requires security enhancements by ACK-CAI.

Execution environment: A machine with kubectl and Helm configured and access to the cluster. Use it directly in Workbench or CloudShell.

1. Create an empty Helm Chart directory.

   mkdir -p ack-cai-vllm-demo
   cd ack-cai-vllm-demo

2. Initialize a Helm Chart for deploying the vLLM service.

   This Helm Chart forcibly schedules the vLLM inference service to confidential computing GPU nodes and uses a CSI plugin to store models in OSS.

   Expand to view Helm Chart initialization script

   # Create template file
   mkdir -p ./templates
   cat <<EOF >templates/vllm.yaml
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv-oss
     namespace: {{ .Release.Namespace }}
     labels:
       alicloud-pvname: pv-oss
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: pv-oss
       volumeAttributes:
         bucket: {{ .Values.oss.bucket }}
         path: {{ .Values.oss.path }}
         url: {{ .Values.oss.url }}
         otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
       nodePublishSecretRef:
         name: oss-secret
         namespace: {{ .Release.Namespace }}
   
   ---
   
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: {{ .Release.Namespace }}
   stringData:
     akId: {{ .Values.oss.akId }}
     akSecret: {{ .Values.oss.akSecret }}
   
   ---
   
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-oss
     namespace: {{ .Release.Namespace }}
   spec:
     accessModes:
       - ReadOnlyMany
     resources:
       requests:
         storage: 5Gi
     selector:
       matchLabels:
         alicloud-pvname: pv-oss
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: cai-vllm-svc
     namespace: {{ .Release.Namespace }}
     {{- if .Values.loadbalancer}}
     {{- if .Values.loadbalancer.aclId }}
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: {{ .Values.loadbalancer.aclId }}
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
     {{- end }}
     {{- end }}
     labels:
       app: cai-vllm
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
     selector:
       app: cai-vllm
     type: LoadBalancer
   
   ---
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: cai-vllm
     namespace: {{ .Release.Namespace }}
     labels:
       app: cai-vllm
       trustiflux.alibaba.com/confidential-computing-mode: "ACK-CAI"
     annotations:
       trustiflux.alibaba.com/ack-cai-options: |
   {{ .Values.caiOptions | indent 6 }}
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node.kubernetes.io/instance-type
               operator: In
               values:
                 - ecs.gn8v-tee.4xlarge
                 - ecs.gn8v-tee.6xlarge
                 - ecs.gn8v-tee-8x.16xlarge
                 - ecs.gn8v-tee-8x.48xlarge
     containers:
       - name: inference-service
         image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference:vllm0.5.4-deepgpu-llm24.7-pytorch2.4.0-cuda12.4-ubuntu22.04
         command:
           - bash
         args: ["-c", "vllm serve /tmp/model --port 8080 --host 0.0.0.0 --served-model-name qwen2.5-3b-instruct --device cuda --dtype auto"]
         ports:
           - containerPort: 8080
         resources:
           limits:
             nvidia.com/gpu: 1  # Request 1 GPU card for this container
         volumeMounts:
           - name: pvc-oss
             mountPath: "/tmp/model"
   
     volumes:
       - name: pvc-oss
         persistentVolumeClaim:
           claimName: pvc-oss
   
   EOF
   
   # Create Helm Chart description file
   cat <<EOF > ./Chart.yaml
   apiVersion: v2
   name: vllm
   description: A test based on vllm for ack-cai
   type: application
   version: 0.1.0
   appVersion: "0.1.0"
   EOF
   
   # Create empty Helm Chart variables file values.yaml
   touch values.yaml
   
   
   

3. Edit the values.yaml file and fill in the environment context.

   Replace <trustee-ip> with the Trustee address, and replace the actual OSS parameter information.

   caiOptions: |
     {
         "cipher-text-volume": "pvc-oss",
         "model-decryption-key-id" : "kbs:///default/aliyun/model-decryption-key",
         "trustee-address": "http://<trustee-ip>:8081/api"
     }
   oss:
     bucket: "conf-ai"                          # Replace with the OSS Bucket name where encrypted models are stored
     path: "/qwen2.5-3b-gocryptfs/"             # Replace with the path of the encrypted model file within the OSS Bucket
     url: "https://oss-cn-beijing-internal.aliyuncs.com"   # Replace with the OSS Endpoint
     akId: "xxxxx"                              # Replace with the Alibaba Cloud AK ID
     akSecret: "xxxxx"                          # Replace with the Alibaba Cloud AK Secret

4. Deploy the vLLM service.

   helm install vllm . -n default

5. Check if the CAI component's Sidecar containers are successfully injected into the Pod.

   kubectl get pod cai-vllm -n default -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}'

   In the expected output, the following 5 containers are displayed, indicating successful injection. Wait until all containers change from PodInitializing to Running, which indicates that the service has started.

   cai-sidecar-attestation-agent   Running
   cai-sidecar-confidential-data-hub       Running
   cai-sidecar-tng Running
   cai-sidecar-cachefs     Running
   inference-service       Running

6. Get and record the access address of the vLLM service.

   kubectl get service cai-vllm-svc -o jsonpath='http://{.status.loadBalancer.ingress[0].ip}:{.spec.ports[0].port}{"\n"}'

   The expected output will return a URL in a similar format (<vllm-ip>:<port>):

   http://182.XX.XX.225:8080

Step 6: Securely access the inference service

In addition to server-side security protection, establish an end-to-end encrypted communication link from client to server. This ensures the security of inference data during transmission. This section starts a TNG security gateway on the client, creating a local proxy that automatically encrypts all requests sent to the vLLM service and decrypts received responses.

Execution environment: Client environment, which is any machine that needs to invoke the vLLM inference service.

1. Start the TNG gateway on the client to establish a secure communication channel.

   The TNG gateway creates a local proxy on the client to encrypt requests sent to the server.

   Replace <IP> with the Trustee address.

   docker run -d \
       --network=host \
       confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product/tng:2.2.4 \
       tng launch --config-content '
         {
           "add_ingress": [
             {
               "http_proxy": {
                 "proxy_listen": {
                   "host": "0.0.0.0",
                   "port": 41000
                 }
               },
               "encap_in_http": {},
               "verify": {
    "as_addr": "http://<trustee-ip>:8081/api/attestation-service/",
                 "policy_ids": [
                   "default"
                 ]
               }
             }
           ]
         }
   '

2. Access the vLLM service through the TNG proxy.

   Replace <vllm-ip>:<port> with the previously obtained access address of the vLLM service.

   # Set the http_proxy environment variable
   export http_proxy=http://127.0.0.1:41000
   
   # Send a curl request
   curl http://<vllm-ip>:<port>/v1/completions \
     -H "Content-type: application/json" \
     -d '{
       "model": "qwen2.5-3b-instruct",
       "prompt": "San Francisco is a",
       "max_tokens": 7,
       "temperature": 0
       }'

Reference information

{
  "cipher-text-volume": "pvc-oss",
  "model-decryption-key-id": "kbs:///default/aliyun/model-decryption-key",
  "trustee-address": "http://<trustee-ip>:8081/api",
  "aa-version": "1.3.1",
  "cdh-version": "1.3.1",
  "tng-version": "2.2.4",
  "cachefs-version": "1.0.7-2.6.1",
  "tdx-ra-enable": true,
  "gpu-ra-enable": true,
  "tng-http-secure-ports": [
    {
      "port": 8080
    }
  ]
}

"tng-http-secure-ports": [
  {
    "port": 8080,
    "allow-insecure-request-regexes": [
      "/api/builtin/.*"
    ]
  }
]