Container Service for Kubernetes:Securely deploy vLLM inference services in an ACK heterogeneous confidential computing cluster

ACK-CAI lets you deploy vLLM inference services in an ACK heterogeneous confidential computing cluster. This provides secure isolation and encrypted protection for your models and data. Key advantages:

• Hardware-level security isolation: Builds a hardware-based TEE using Intel® TDX and NVIDIA GPU TEE, ensuring the confidentiality and integrity of models and data during computation.

• Trusted key distribution: Uses a remote attestation mechanism to strictly verify the integrity of the execution environment. Only after successful verification does a separate trustee service release the model decryption key to the trusted environment.

• End-to-end data encryption: Establishes an encrypted channel from the client to the server through a Trusted Network Gateway (TNG), protecting inference requests and responses during transmission.

• Non-intrusive for applications: Automatically injects security components into pods using a Kubernetes webhook. You can enable confidential computing capabilities for your application with a simple annotation, requiring no changes to your business code or container images.

How it works

ACK-CAI enables transparent confidential computing capabilities by dynamically injecting a set of sidecar containers called Trustiflux into your application pods. The core security mechanism is based on remote attestation, which ensures that models and data are only accessed within a verified, trusted environment.

Process and environment guide

Deploying and accessing a secure vLLM inference service involves the following stages:

Step 1: Prepare the encrypted model

This step covers how to encrypt your model data and upload it to OSS in preparation for secure distribution.

Execution environment: To ensure security, perform these steps on a temporary, isolated ECS instance. For optimal performance, this instance should be in the same region as your OSS bucket to leverage high-speed internal network uploads.

The model files are large, and this process can be time-consuming. To quickly test the solution, you can skip this step. Use the sample encrypted model file and proceed to Step 2: Deploy the trustee remote attestation service.

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

Step 2: Deploy the trustee remote attestation service

Following Zero Trust principles, any confidential computing environment must be verified before it can access sensitive data, such as the model decryption keys.

The standalone trustee service, which you will deploy in this step, acts as the central authority for this verification process. It is responsible for:

• Verifying the execution environment of the model and inference service.

• Ensuring that model decryption keys are released only to a verified, trusted environment.

• Enabling clients to confirm the trustworthiness of the service when initiating an inference request.

Execution environment: A dedicated, separate server deployed outside the ACK cluster, such as an ECS instance or an on-premises server.

Step 3: Configure the ACK confidential computing cluster

In this step, you will build the underlying infrastructure: an ACK cluster with ecs.gn8v-tee instances that provide both Intel TDX and NVIDIA TEE capabilities as worker nodes.

Execution environment: The ECS and ACK consoles (for creating clusters, node pools, and ECS instances), and the shell environment of the created ecs.gn8v-tee instance (for installing drivers).

1. Create an ACK managed Pro cluster in the China (Beijing) region.

2. Create a node pool for the cluster to manage the confidential computing instances.

   • vSwitch: Select a virtual switch in Zone L of the China (Beijing) region.

   • Scaling Mode: Keep the default configurations. Do not enable auto scaling.

   • Instance Type: ecs.gn8v-tee.4xlarge or a higher specification.

   • Operating System: Alibaba Cloud Linux 3.2104 LTS 64-bit.

   • System Disk: 100 GiB or larger.

   • Expected Nodes: The initial number of nodes in the node pool. Keep the default configuration of 0.

   • Node Labels: Add a label (Key: ack.aliyun.com/nvidia-driver-version, Value: 550.144.03) to specify the NVIDIA driver version.

3. Create an Elastic GPU Service (EGS) confidential computing instance to serve as a cluster node. See Create an instance on the Custom Launch tab.

   • Region: China (Beijing).

   • Network and Zone: The VPC must be the same as the cluster's VPC. This example uses one in Zone L.

   • Instance: ecs.gn8v-tee.4xlarge or a higher specification.

     The gn8v-tee instance types have CPU and GPU confidential computing features enabled by default. You do not need to select confidential VM.

   • Image: Alibaba Cloud Linux 3.2104 LTS 64-bit.

4. Log on to the created EGS instance and install the NVIDIA driver and CUDA toolkit.

5. Add the EGS instance to the created node pool. Select Manual as the method for adding the instance. See Add existing ECS instances.

Step 4: Deploy the ACK-CAI component

The ACK-CAI component includes a webhook controller that automatically injects the necessary sidecar containers into pods based on their annotations. These sidecars handle remote attestation, model decryption, and secure communication.

Execution environment: ACK console.

1. Log on to the ACK console. In the navigation pane on the left, click Clusters.

2. On the Clusters page, find the cluster you want and click its name. In the left-side navigation pane, choose Applications > Helm.

3. Click Deploy and install the latest version of ack-cai.

   In the Parameters step, change the tag to 1.1.1 in the YAML template.

   You can now view the deployment status in the Helm chart list.

Step 5: Deploy the vLLM inference service

Deploy the vLLM service using Helm and add the specific annotation to enable confidential computing protection.

Execution environment: A machine with kubectl and Helm configured and able to access the cluster. You can use Workbench or CloudShell.

1. Create a new folder for the Helm chart.

   mkdir -p ack-cai-vllm-demo
   cd ack-cai-vllm-demo

2. Initialize a Helm chart to deploy the vLLM service.

   This Helm chart configures the vLLM inference service with node affinity to ensure it runs only on confidential computing GPU-accelerated nodes. It also uses a CSI plugin to mount an OSS bucket for model storage.

   Expand to view the Helm chart initialization script

   # Create the template file
   mkdir -p ./templates
   cat <<EOF >templates/vllm.yaml
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv-oss
     namespace: {{ .Release.Namespace }}
     labels:
       alicloud-pvname: pv-oss
   spec:
     capacity:
       storage: 5Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: pv-oss
       volumeAttributes:
         bucket: {{ .Values.oss.bucket }}
         path: {{ .Values.oss.path }}
         url: {{ .Values.oss.url }}
         otherOpts: "-o umask=022 -o max_stat_cache_size=0 -o allow_other"
       nodePublishSecretRef:
         name: oss-secret
         namespace: {{ .Release.Namespace }}
   
   ---
   
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: {{ .Release.Namespace }}
   stringData:
     akId: {{ .Values.oss.akId }}
     akSecret: {{ .Values.oss.akSecret }}
   
   ---
   
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-oss
     namespace: {{ .Release.Namespace }}
   spec:
     accessModes:
       - ReadOnlyMany
     resources:
       requests:
         storage: 5Gi
     selector:
       matchLabels:
         alicloud-pvname: pv-oss
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: cai-vllm-svc
     namespace: {{ .Release.Namespace }}
     {{- if .Values.loadbalancer}}
     {{- if .Values.loadbalancer.aclId }}
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: {{ .Values.loadbalancer.aclId }}
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
     {{- end }}
     {{- end }}
     labels:
       app: cai-vllm
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
     selector:
       app: cai-vllm
     type: LoadBalancer
   
   ---
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: cai-vllm
     namespace: {{ .Release.Namespace }}
     labels:
       app: cai-vllm
       trustiflux.alibaba.com/confidential-computing-mode: "ACK-CAI"
     annotations:
       trustiflux.alibaba.com/ack-cai-options: |
   {{ .Values.caiOptions | indent 6 }}
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node.kubernetes.io/instance-type
               operator: In
               values:
                 - ecs.gn8v-tee.4xlarge
                 - ecs.gn8v-tee.6xlarge
                 - ecs.gn8v-tee-8x.16xlarge
                 - ecs.gn8v-tee-8x.48xlarge
     containers:
       - name: inference-service
         image: egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/llm-inference:vllm0.5.4-deepgpu-llm24.7-pytorch2.4.0-cuda12.4-ubuntu22.04
         command:
           - bash
         args: ["-c", "vllm serve /tmp/model --port 8080 --host 0.0.0.0 --served-model-name qwen2.5-3b-instruct --device cuda --dtype auto"]
         ports:
           - containerPort: 8080
         resources:
           limits:
             nvidia.com/gpu: 1  # Request 1 GPU for this container
         volumeMounts:
           - name: pvc-oss
             mountPath: "/tmp/model"
   
     volumes:
       - name: pvc-oss
         persistentVolumeClaim:
           claimName: pvc-oss
   
   EOF
   
   # Create the Helm chart description file
   cat <<EOF > ./Chart.yaml
   apiVersion: v2
   name: vllm
   description: A test based on vllm for ack-cai
   type: application
   version: 0.1.0
   appVersion: "0.1.0"
   EOF
   
   # Create a variables file for the Helm chart named values.yaml
   touch values.yaml
   
   
   

3. Edit the values.yaml file to provide your environment-specific information.

   Replace <trustee-ip> with the trustee address and replace the OSS configurations with your actual values.

   caiOptions: |
     {
         "cipher-text-volume": "pvc-oss",
         "model-decryption-key-id" : "kbs:///default/aliyun/model-decryption-key",
         "trustee-address": "http://<trustee-ip>:8081/api"
     }
   oss:
     bucket: "conf-ai"                          # Replace with the name of the OSS bucket that stores the encrypted model.
     path: "/qwen2.5-3b-gocryptfs/"             # Replace with the path to the encrypted model file in the OSS Bucket.
     url: "https://oss-cn-beijing-internal.aliyuncs.com"   # Replace with the OSS endpoint.
     akId: "xxxxx"                              # Replace with your Alibaba Cloud AccessKey ID.
     akSecret: "xxxxx"                          # Replace with your Alibaba Cloud AccessKey secret.

4. Deploy the vLLM service using Helm.

   helm install vllm . -n default

5. Verify that the CAI component's sidecar containers have been successfully injected into the pod.

   kubectl get pod cai-vllm -n default -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{"\t"}{range .state.running}Running{end}{.state.*.reason}{"\n"}{end}'

   The expected output shows the following five containers, meaning the injection succeeded. Wait for all containers to change from PodInitializing to Running. This indicates that the service has started.

   cai-sidecar-attestation-agent   Running
   cai-sidecar-confidential-data-hub       Running
   cai-sidecar-tng Running
   cai-sidecar-cachefs     Running
   inference-service       Running

6. Get and record the vLLM service endpoint.

   kubectl get service cai-vllm-svc -o jsonpath='http://{.status.loadBalancer.ingress[0].ip}:{.spec.ports[0].port}{"\n"}'

   The expected output is a URL in the format <vllm-ip>:<port>.

   http://182.XX.XX.225:8080

Step 6: Securely access the inference service

To ensure end-to-end security, you must use the TNG client gateway to proxy your requests. The proxy automatically encrypts all requests sent to the vLLM service and decrypts the responses.

Execution environment: The client machine from which you want to call the inference service.

1. Start the TNG gateway on the client to establish a secure communication channel.

   The TNG gateway creates a local proxy on the client to encrypt requests sent to the server.

   Replace <IP> with the trustee address.

   docker run -d \
       --network=host \
       confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product/tng:2.2.4 \
       tng launch --config-content '
         {
           "add_ingress": [
             {
               "http_proxy": {
                 "proxy_listen": {
                   "host": "0.0.0.0",
                   "port": 41000
                 }
               },
               "encap_in_http": {},
               "verify": {
    "as_addr": "http://<trustee-ip>:8081/api/attestation-service/",
                 "policy_ids": [
                   "default"
                 ]
               }
             }
           ]
         }
   '

2. Access the vLLM service through the TNG proxy.

   Replace <vllm-ip>:<port> with the endpoint of the vLLM service you obtained earlier.

   # Set the http_proxy environment variable
   export http_proxy=http://127.0.0.1:41000
   
   # Send a curl request
   curl http://<vllm-ip>:<port>/v1/completions \
     -H "Content-type: application/json" \
     -d '{
       "model": "qwen2.5-3b-instruct",
       "prompt": "San Francisco is a",
       "max_tokens": 7,
       "temperature": 0
       }'

Reference

{
  "cipher-text-volume": "pvc-oss",
  "model-decryption-key-id": "kbs:///default/aliyun/model-decryption-key",
  "trustee-address": "http://<trustee-ip>:8081/api",
  "aa-version": "1.3.1",
  "cdh-version": "1.3.1",
  "tng-version": "2.2.4",
  "cachefs-version": "1.0.7-2.6.1",
  "tdx-ra-enable": true,
  "gpu-ra-enable": true,
  "tng-http-secure-ports": [
    {
      "port": 8080
    }
  ]
}

"tng-http-secure-ports": [
  {
    "port": 8080,
    "allow-insecure-request-regexes": [
      "/api/builtin/.*"
    ]
  }
]