Container Service for Kubernetes:Securely Deploy vLLM Inference Services in ACK Heterogeneous Confidential Computing Clusters

How it works

ACK-CAI provides confidential computing capabilities by injecting a set of sidecar containers called Trustiflux into application pods. Security is enforced through remote attestation, ensuring that models and inference data are accessible only inside a verified trusted environment.

Prerequisites

Before you begin, make sure you have:

• An Alibaba Cloud account with permissions to create ACK clusters and Elastic Compute Service (ECS) instances

• A dedicated server (ECS instance or on-premises) to deploy the Trustee remote attestation service, with public network access and port 8081 open

• A machine with kubectl and Helm installed, with access to the ACK cluster API server

• Docker installed on the client machine used to access the inference service

• (Optional) An Object Storage Service (OSS) bucket in the China (Beijing) region for storing encrypted model files

Deployment overview

Deploying a secure vLLM inference service involves six steps across different environments.

Tip: Step 1 (encrypting the model) is optional. If you want to test the solution quickly, skip to Step 2 and use the pre-encrypted sample models stored in a public-read OSS bucket.

Step 1: Prepare encrypted models

This step encrypts your model and uploads the ciphertext to OSS, preparing it for secure remote distribution.

Run these commands on a temporary ECS instance in the same region as your OSS bucket to maximize upload speed over the private network.

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

Step 2: Deploy the Trustee remote attestation service

Following the zero trust principle, every confidential computing environment must pass verification before gaining access to sensitive resources such as model decryption keys. Trustee acts as the root of trust: it verifies the runtime environment and distributes keys only after successful attestation.

Step 3: Configure the ACK confidential computing cluster

This step creates an ACK cluster and adds ecs.gn8v-tee instances—which have both Intel TDX and NVIDIA GPU TEE capabilities enabled by default—as worker nodes.

gn8v-tee instance types have CPU and GPU confidential computing enabled by default. No additional configuration is needed to enable the confidential VM feature.

1. Create an ACK managed cluster Pro edition in the China (Beijing) region. For instructions, see Create an ACK managed cluster.

2. Create a node pool to manage the confidential computing instances. For instructions, see Create and manage node pools. Use the following settings:

   Setting	Value
   vSwitch	Select the virtual switch in China (Beijing) Zone L
   Scaling mode	Keep the default (do not enable automatic elastic scaling)
   Instance type	ecs.gn8v-tee.4xlarge or above
   Operating system	Alibaba Cloud Linux 3.2104 LTS 64-bit
   System disk	100 GiB or more
   Expected number of nodes	0 (default)
   Node labels	Key: ack.aliyun.com/nvidia-driver-version, Value: 550.144.03

3. Create an ecs.gn8v-tee instance to use as a worker node. For instructions, see Custom purchase instances. Use the following settings:

   Setting	Value
   Region	China (Beijing)
   Network and zone	Same VPC as the cluster, Zone L
   Instance type	ecs.gn8v-tee.4xlarge or above
   Image	Alibaba Cloud Linux 3.2104 LTS 64-bit

4. Log in to the ecs.gn8v-tee instance and install the NVIDIA drivers and CUDA toolkit. For instructions, see Install NVIDIA drivers and CUDA toolkit.

5. Add the instance to the node pool using manual addition. For instructions, see Add existing nodes.

Step 4: Deploy ACK-CAI components

ACK-CAI uses a Kubernetes webhook controller to automatically inject Trustiflux sidecar containers into pods. These sidecars provide remote attestation, model decryption, and secure communication—all without modifying your application.

1. Log in to the Container Service Management Console. In the left navigation pane, click Clusters.

2. On the Clusters page, click your cluster name. In the left navigation pane, click Applications > Helm.

3. Click Create and follow the on-screen prompts to install the latest version of ACK-CAI. After installation, verify the deployment status in the Helm chart list.

Step 5: Deploy the vLLM model inference service

This step deploys the vLLM inference service using Helm. The deployment uses an annotation on the pod to signal ACK-CAI to inject the Trustiflux sidecar, enabling confidential computing protection.

Run these commands on a machine with kubectl and Helm configured and connected to the ACK cluster API server. You can also use Workbench or CloudShell.

1. Create the Helm chart directory:

   mkdir -p ack-cai-vllm-demo
   cd ack-cai-vllm-demo

2. Initialize the Helm chart. The chart schedules the vLLM pod on confidential computing GPU nodes and uses the CSI plugin to mount the encrypted model from OSS.

   Expand to view Helm Chart initialization script

   # Create the templates directory and vLLM manifest
   mkdir -p ./templates
   cat <<EOF >templates/vllm.yaml
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv-oss
     namespace: {{ .Release.Namespace }}
     labels:
       alicloud-pvname: pv-oss
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: pv-oss
       volumeAttributes:
         bucket: {{ .Values.oss.bucket }}
         path: {{ .Values.oss.path }}
         url: {{ .Values.oss.url }}
         otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
       nodePublishSecretRef:
         name: oss-secret
         namespace: {{ .Release.Namespace }}
   
   ---
   
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: {{ .Release.Namespace }}
   stringData:
     akId: {{ .Values.oss.akId }}
     akSecret: {{ .Values.oss.akSecret }}
   
   ---
   
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-oss
     namespace: {{ .Release.Namespace }}
   spec:
     accessModes:
       - ReadOnlyMany
     resources:
       requests:
         storage: 5Gi
     selector:
       matchLabels:
         alicloud-pvname: pv-oss
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: cai-vllm-svc
     namespace: {{ .Release.Namespace }}
     {{- if .Values.loadbalancer}}
     {{- if .Values.loadbalancer.aclId }}
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: {{ .Values.loadbalancer.aclId }}
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
     {{- end }}
     {{- end }}
     labels:
       app: cai-vllm
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
     selector:
       app: cai-vllm
     type: LoadBalancer
   
   ---
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: cai-vllm
     namespace: {{ .Release.Namespace }}
     labels:
       app: cai-vllm
       trustiflux.alibaba.com/confidential-computing-mode: "ACK-CAI"
     annotations:
       trustiflux.alibaba.com/ack-cai-options: |
   {{ .Values.caiOptions | indent 6 }}
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node.kubernetes.io/instance-type
               operator: In
               values:
                 - ecs.gn8v-tee.4xlarge
                 - ecs.gn8v-tee.6xlarge
                 - ecs.gn8v-tee-8x.16xlarge
                 - ecs.gn8v-tee-8x.48xlarge
     containers:
       - name: inference-service
         image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference:vllm0.5.4-deepgpu-llm24.7-pytorch2.4.0-cuda12.4-ubuntu22.04
         command:
           - bash
         args: ["-c", "vllm serve /tmp/model --port 8080 --host 0.0.0.0 --served-model-name qwen2.5-3b-instruct --device cuda --dtype auto"]
         ports:
           - containerPort: 8080
         resources:
           limits:
             nvidia.com/gpu: 1  # Request 1 GPU card for this container
         volumeMounts:
           - name: pvc-oss
             mountPath: "/tmp/model"
   
     volumes:
       - name: pvc-oss
         persistentVolumeClaim:
           claimName: pvc-oss
   
   EOF
   
   # Create the Helm chart description file
   cat <<EOF > ./Chart.yaml
   apiVersion: v2
   name: vllm
   description: A test based on vllm for ack-cai
   type: application
   version: 0.1.0
   appVersion: "0.1.0"
   EOF
   
   # Create an empty values file
   touch values.yaml

3. Edit values.yaml with your environment configuration. Replace <trustee-ip> with the Trustee server address and fill in your OSS details.

   caiOptions: |
     {
         "cipher-text-volume": "pvc-oss",
         "model-decryption-key-id" : "kbs:///default/aliyun/model-decryption-key",
         "trustee-address": "http://<trustee-ip>:8081/api"
     }
   oss:
     bucket: "conf-ai"                          # OSS bucket name where the encrypted model is stored
     path: "/qwen2.5-3b-gocryptfs/"             # Path to the encrypted model within the bucket
     url: "https://oss-cn-beijing-internal.aliyuncs.com"   # OSS endpoint
     akId: "xxxxx"                              # Alibaba Cloud AccessKey ID
     akSecret: "xxxxx"                          # Alibaba Cloud AccessKey Secret

4. Deploy the vLLM service:

   helm install vllm . -n default

5. Verify that the ACK-CAI sidecar containers are injected into the pod:

   kubectl get pod cai-vllm -n default -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}'

   The output should list all five containers. Wait until they all show Running:

   cai-sidecar-attestation-agent   Running
   cai-sidecar-confidential-data-hub       Running
   cai-sidecar-tng Running
   cai-sidecar-cachefs     Running
   inference-service       Running

6. Get the vLLM service address:

   kubectl get service cai-vllm-svc -o jsonpath='http://{.status.loadBalancer.ingress[0].ip}:{.spec.ports[0].port}{"\n"}'

   The output returns a URL in the format http://<vllm-ip>:8080. Save this address for Step 6.

Step 6: Access the inference service securely

Before sending inference requests, start a TNG Client on your machine to establish an encrypted channel. The TNG Client creates a local proxy on port 41000 that encrypts all outbound requests and decrypts incoming responses.

1. Start the TNG Client:

   Replace <trustee-ip> with the Trustee server address.

   docker run -d \
       --network=host \
       confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product/tng:2.2.4 \
       tng launch --config-content '
         {
           "add_ingress": [
             {
               "http_proxy": {
                 "proxy_listen": {
                   "host": "0.0.0.0",
                   "port": 41000
                 }
               },
               "encap_in_http": {},
               "verify": {
                 "as_addr": "http://<trustee-ip>:8081/api/attestation-service/",
                 "policy_ids": [
                   "default"
                 ]
               }
             }
           ]
         }
   '

2. Send a request through the TNG proxy. Replace <vllm-ip>:<port> with the service address obtained in Step 5.

   # Route requests through the local TNG proxy
   export http_proxy=http://127.0.0.1:41000
   
   curl http://<vllm-ip>:<port>/v1/completions \
     -H "Content-type: application/json" \
     -d '{
       "model": "qwen2.5-3b-instruct",
       "prompt": "San Francisco is a",
       "max_tokens": 7,
       "temperature": 0
       }'

Reference

{
  "cipher-text-volume": "pvc-oss",
  "model-decryption-key-id": "kbs:///default/aliyun/model-decryption-key",
  "trustee-address": "http://<trustee-ip>:8081/api",
  "aa-version": "1.3.1",
  "cdh-version": "1.3.1",
  "tng-version": "2.2.4",
  "cachefs-version": "1.0.7-2.6.1",
  "tdx-ra-enable": true,
  "gpu-ra-enable": true,
  "tng-http-secure-ports": [
    {
      "port": 8080
    }
  ]
}

"tng-http-secure-ports": [
  {
    "port": 8080,
    "allow-insecure-request-regexes": [
      "/api/builtin/.*"
    ]
  }
]