Elastic GPU Service:Build a secure LLM inference environment on a heterogeneous confidential computing instance

1. Download a model

Before deploying a model, you need to encrypt the model and upload it to cloud storage. The key for decrypting the model will be hosted by KMS, controlled by the remote attestation service. Perform model encryption operations in a local or trusted environment. This solution shows how to deploy Qwen2.5-3B-Instruct as an example.

Run the following command in the terminal use ModelScope tool to download Qwen2.5-3B-Instruct(requires Python 3.9 or higher).

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

The command will download the model to ~/.cache/modelscope/hub/models/Qwen/Qwen2.5-3B-Instruct/.

2. Encrypt the model

The following two encryption methods are supported. This topic uses Gocryptfs as an example.

• Gocryptfs: An encryption pattern based on AES-256-GCM that is compliant with the open source Gocryptfs standard.

• Sam: An Alibaba Cloud trusted AI model encryption format. It protects model confidentiality and prevents license content from being tampered with or used illegally.

3. Upload the model

Prepare an OSS Bucket in the same region where you will deploy the heterogeneous instance. Upload the encrypted model to Alibaba Cloud OSS. This lets you pull and deploy the model from the heterogeneous instance later.

Take OSS as an example. You can create a bucket and a directory named qwen-encrypted, such as oss://examplebucket/qwen-encrypted/. For more information, see Quick Start for the console. Because the model file is large, we recommend using ossbrowser to upload the encrypted model to this directory.

1. Create an Alibaba Cloud KMS instance as the key storage backend

1. Go to the KMS console, choose Resource > Instances in the left-side navigation pane. Then, create and start an instance on the Software Key Management tab. When starting the instance, select the same VPC as the ACK cluster. For more information, see Purchase and enable a KMS instance.

   Wait for about 10 minutes for the instance to start.

2. After the instance is started, choose Resource > Keys in the left-side navigation pane. Then, create a customer master key for the instance on the Keys page. For more information, see Step 1: Create a software-protected key.

3. Choose Application Access > AAPs in the left-side navigation pane. Then, create an application access point (AAP) for the instance on the AAPs page. Set Scope to the created KMS instance. For more information, see Mode 1: Quick creation.

   After the AAP is created, your browser automatically downloads a ClientKey***.zip file. This zip file contains the following files after extraction:

   • ClientKeyContent: The default file name is clientKey_****.json.

   • ClientKeyPassword: The default file name is clientKey_****_Password.txt.

4. Go to Resource > Instances, click the KMS instance name. Then, in the Basic Information section, click Download next to Instance CA Certificate to export the public key certificate file PrivateKmsCA_***.pem of the KMS instance.

2. Create an ACK service cluster and install the csi-provisioner component

1. Go to the Create Cluster page to create an ACK Serverless cluster. Take note of the following parameters. For more parameter settings, see Create an ACK Serverless cluster.

   1. Cluster Configurations: Configure the following parameters and click Next: Component Configurations.

      Key item	Description
      VPC	Select Use Existing and select Configure SNAT For VPC. Otherwise, you cannot pull the Trustee image.
      vSwitch	Make sure that at least two vSwitches are created in the existing VPC. Otherwise, you cannot expose the public ALB.

   2. Component Configurations: Configure the following parameters and click Next: Confirm.

      Key item	Description
      Service Discovery	Select CoreNDS.
      Ingress	Select ALB Ingress. For Gateway Source, select New and select two vSwitches.

   3. Confirm: Confirm the configuration information and terms of service, and click Create Cluster.

2. After the cluster is created, install the csi-provisioner (Managed) component. For more information, see Manage components.

3. Deploy the Trustee remote attestation service in the ACK cluster

1. First, connect to the cluster over the Internet or an internal network. For more information, see Connect to a cluster.

2. Upload the downloaded client key content (clientKey_****.json), key password (clientKey_****_Password.txt), and CA certificate (PrivateKmsCA_***.pem) to the environment connected to the ACK Serverless cluster, and run the following command to deploy Trustee using Alibaba Cloud KMS as the key storage backend.

   # Install the plugin
   helm plugin install https://github.com/AliyunContainerService/helm-acr
   
   helm repo add trustee acr://trustee-chart.cn-hangzhou.cr.aliyuncs.com/trustee/trustee
   helm repo update
   
   export DEPLOY_RELEASE_NAME=trustee
   export DEPLOY_NAMESPACE=default
   export TRUSTEE_CHART_VERSION=1.0.0
   
   # Set the region information of the ACK cluster, such as cn-hangzhou
   export REGION_ID=cn-hangzhou
   
   # KMS instance information exported earlier 
   # Replace with your KMS instance ID
   export KMS_INSTANCE_ID=kst-hzz66a0*******e16pckc
   # Replace with the path to your KMS instance application identity secret 
   export KMS_CLIENT_KEY_FILE=/path/to/clientKey_KAAP.***.json
   # Replace with the path to your KMS instance secret security token 
   export KMS_PASSWORD_FILE=/path/to/clientKey_KAAP.***_Password.txt
   # Replace with the path to your KMS instance CA certificate
   export KMS_CERT_FILE=/path/to/PrivateKmsCA_kst-***.pem
   
   helm install ${DEPLOY_RELEASE_NAME} trustee/trustee \
     --version ${TRUSTEE_CHART_VERSION} \
     --set regionId=${REGION_ID} \
     --set kbs.aliyunKms.enabled=true \
     --set kbs.aliyunKms.kmsIntanceId=${KMS_INSTANCE_ID} \
     --set-file kbs.aliyunKms.clientKey=${KMS_CLIENT_KEY_FILE} \
     --set-file kbs.aliyunKms.password=${KMS_PASSWORD_FILE} \
     --set-file kbs.aliyunKms.certPem=${KMS_CERT_FILE} \
     --namespace ${DEPLOY_NAMESPACE}

   Note

   The first command (helm plugin install...) may take a long time to execute. If the installation fails, you can first uninstall the plugin by running helm plugin uninstall cm-push, and then run the plugin installation command again.

   Sample output:

   NAME: trustee
   LAST DEPLOYED: Tue Feb 25 18:55:33 2025
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None

3. Run the following command in the environment connected to the ACK Serverless cluster to obtain the access address of Trustee.

   export TRUSTEE_URL=http://$(kubectl get AlbConfig alb -o jsonpath='{.status.loadBalancer.dnsname}')/${DEPLOY_RELEASE_NAME}
   echo ${TRUSTEE_URL}

   Sample output: http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/trustee.

4. Run the following command in the environment connected to the ACK Serverless cluster to test the connectivity of the Trustee service.

   cat << EOF | curl -k -X POST ${TRUSTEE_URL}/kbs/v0/auth -H 'Content-Type: application/json' -d @-
   {
       "version":"0.1.0",
       "tee": "tdx",
       "extra-params": "foo"
   }
   EOF

   If Trustee is running normally, the expected output is:

   {"nonce":"PIDUjUxQdBMIXz***********IEysXFfUKgSwk=","extra-params":""}

4. Create a secret to store the model decryption key

The model decryption key managed by Trustee is stored in KMS. The key can only be accessed after the remote attestation service verifies the target environment.

Go to the Key Management Service console. In the navigation pane on the left, choose Resource > Secrets. On the Generic Secrets tab, click Create Generic Secret. Note the following key configurations:

• Secret Name: Enter a custom name for the secret. This name is used as an index for the key. For example, model-decryption-key.

• Set Secret Value: Enter the key that was used to encrypt the model. For example, 0Bn4Q1wwY9fN3P. Use your actual key.

• Encryption CMK: Select the master key that you created in the previous step.

1. Configure access permissions to OSS

During deployment, the instance needs to access the OSS bucket that stores the model ciphertext and access Trustee to obtain the model decryption key. Therefore, you must configure access permissions for the instance to the OSS and Trustee services.

1. Log on to the OSS console.

2. Click Buckets, and then click the target bucket name.

3. In the navigation pane on the left, choose Permission Control > Bucket Policy.

4. On the Bucket Policy tab, click Authorize. In the panel that appears, grant the Bucket permission to the public IP address of the heterogeneous confidential computing instance.

5. Click OK.

2. Configure access permissions to Trustee

• Go to the Application Load Balancer (ALB) console, select Access Control on the left. Then click Create ACL, and add the IP addresses or CIDR blocks that need to access Trustee as IP entries. For more information, see Access control.

  You must add the following addresses or address ranges:

  • The public IP address of the VPC that is bound during the heterogeneous service deployment.

  • The egress IP address of the inference client.

• Run the following command to obtain the ID of the ALB instance that is used by the Trustee instance on the cluster.

  kubectl get ing --namespace ${DEPLOY_NAMESPACE} kbs-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | cut -d'.' -f1 | sed 's/[^a-zA-Z0-9-]//g'

  The expected output is as follows:

  alb-llcdzbw0qivhk0****

• In the navigation pane on the left of the ALB console, choose ALB > Instances. In the region where the cluster is located, search for the ALB instance that you obtained in the previous step and click the instance ID to go to the instance details page. At the bottom of the page, in the Instance Information area, click Disable Configuration Read-only Mode.

• Switch to the Listener tab, click Enable in the Access Control column of the target listener instance, and configure the whitelist with the access control policy group that you created in the previous step.

1. Prepare the configuration file

1. Remotely connect to the heterogeneous instance. For more information, see Log on to a Linux instance using Workbench.

2. Run the following command to open the ~/cai.env configuration file.

   vi ~/cai.env

3. Press the i key to enter insert mode and paste the prepared configuration information into the file.

   # Configurations for model decryption
   TRUSTEE_URL="http://alb-xxxxxxxxx.cn-beijing.alb.aliyuncsslb.com/xxxx/" # URL of the Trustee service
   MODEL_KEY_ID="kbs:///default/aliyun/model-decryption-key" # The resource ID of the model decryption key configured on Trustee. For example, the ID for the secret named model-decryption-key is kbs:///default/aliyun/model-decryption-key
   
   # Configurations for encrypted model distribution
   MODEL_OSS_BUCKET_PATH="<bucket-name>:<model-path>" # The path of the model ciphertext stored in Alibaba Cloud OSS. For example, conf-ai:/qwen3-32b-gocryptfs/
   MODEL_OSS_ENDPOINT="https://oss-cn-beijing-internal.aliyuncs.com" # The OSS URL where the model ciphertext is stored
   # MODEL_OSS_ACCESS_KEY_ID="" # The AccessKey ID used to access the OSS bucket.
   # MODEL_OSS_SECRET_ACCESS_KEY="" # The AccessKey secret used to access the OSS bucket.
   # MODEL_OSS_SESSION_TOKEN="" # The session token used to access the OSS bucket. This is required only when you use STS credentials to access OSS.
   MODEL_ENCRYPTION_METHOD="gocryptfs" # The encryption method used. Valid values: "gocryptfs" and "sam". Default value: "gocryptfs".
   MODEL_MOUNT_POINT=/tmp/model # The mount point for the decrypted model plaintext. The model inference service can load the model from this path.
   
   # Configurations for large model service communication encryption
   MODEL_SERVICE_PORT=8080 # The TCP port used by the service. Communication on this port is transparently encrypted.
   
   # Configurations for P2P model distribution acceleration. You can ignore this by default.
   TRUSTEE_POLICY="default"
   # MODEL_SHARING_PEER="172.30.24.146 172.30.24.144"

4. Enter :wq and press Enter to save the file and exit the editor.

2. Deploy the inference service

1. Configure access permissions for the client environment to the heterogeneous confidential computing instance and Trustee

In this topic, the client environment accesses the inference service that is deployed in the heterogeneous confidential computing instance through its public IP address. Therefore, you must add a rule to the security group of the heterogeneous confidential computing instance to allow access from the client environment. In addition, during the process of establishing a secure channel with the server, Trustee is required for remote attestation of the heterogeneous confidential computing instance. Therefore, you must also add the public IP address of the client environment to the access control list of Trustee. The procedure is as follows.

1. Add a rule to the security group of the heterogeneous confidential computing instance to allow access from the client. For more information, see Manage security group rules.

2. Add the public IP address of the client environment to the access control of Trustee.

   You have already created an access control policy group and added it to the whitelist in Step 4: Configure access permissions for the heterogeneous instance to OSS and Trustee. Therefore, you do not need to create a new one. You can add the public IP address of the client to the existing access control policy group.

2. Deploy the Trusted Network Gateway client on the client instance

Trusted Network Gateway (TNG) is a network component that is designed for confidential computing scenarios. It can act as a daemon process that is responsible for establishing secure communication channels and transparently encrypting and decrypting network traffic to and from confidential instances to achieve end-to-end data security. It also lets you flexibly control the traffic encryption and decryption process based on your needs without modifying existing applications.

1. Log on to a Linux instance using Workbench.

2. Deploy the Trusted Network Gateway client.

   Deploy using docker

   1. Install Docker on the client instance. For more information, see Install and use Docker and Docker Compose.

   2. Run the following command to deploy the Trusted Network Gateway client using Docker.

      Important

      You must modify the value of the as_addr field to ${trsutee_url}/as/ based on the Trustee service that you previously deployed.

      docker run --rm \
          --network=host \
          confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product/tng:2.2.1 \
          tng launch --config-content '
              {
                  "add_ingress": [
                      {
                          "http_proxy": {
                              "proxy_listen": {
                                  "host": "127.0.0.1",
                                  "port": 41000
                              }
                          },
                          "verify": {
                              "as_addr": "http://alb-xxxxxxxxxxxxxxxxxx.cn-shanghai.alb.aliyuncsslb.com/cai-test/as/",
                              "policy_ids": [
                                  "default"
                              ]
                          }
                      }
                  ]
              }
          '

   Deploy using a binary file

   1. Visit TNG · GitHub to obtain the download address of the binary installation package that matches the client instance architecture.

   2. Run the following command to download the Trusted Network Gateway client binary file. The following example uses tng-v2.2.1.x86_64-unknown-linux-gnu.tar.gz. Replace it with the actual filename.

      wget https://github.com/inclavare-containers/TNG/releases/download/v2.2.1/tng-v2.2.1.x86_64-unknown-linux-gnu.tar.gz

   3. Run the following command to decompress the downloaded binary compressed file and grant executable permissions to the file.

      tar -zxvf tng-v2.2.1.x86_64-unknown-linux-gnu.tar.gz
      chmod +x tng

   4. Run the following command to run the Trusted Network Gateway client.

      Important

      You must modify the value of the as_addr field to ${trsutee_url}/as/ based on the Trustee service that you previously deployed.

      ./tng launch --config-content '
              {
                  "add_ingress": [
                      {
                          "http_proxy": {
                              "proxy_listen": {
                                  "host": "127.0.0.1",
                                  "port": 41000
                              }
                          },
                          "verify": {
                              "as_addr": "http://alb-xxxxxxxxxxxxxxxxxx.cn-shanghai.alb.aliyuncsslb.com/cai-test/as/",
                              "policy_ids": [
                                  "default"
                              ]
                          }
                      }
                  ]
              }
          '

3. Configure the HTTP proxy service for processes on the client instance

After the deployment is successful, the Trusted Network Gateway client runs in the foreground and creates an HTTP proxy service that is based on the HTTP CONNECT protocol on 127.0.0.1:41000. When you connect an application to this proxy, the traffic is encrypted by the Trusted Network Gateway client and sent to the vLLM service through a trusted channel.

You can configure the HTTP proxy service for processes on the client instance in the following two ways.

export http_proxy=http://127.0.0.1:41000
export https_proxy=http://127.0.0.1:41000
export ftp_proxy=http://127.0.0.1:41000
export rsync_proxy=http://127.0.0.1:41000

export all_proxy=http://127.0.0.1:41000

4. Access the inference service from the client instance

You can access the inference service in the heterogeneous confidential computing instance by running the curl command in the client environment.

1. Open a new terminal window for the client instance.

2. Run the following command to access the inference service.

   Note

   • You must replace <heterogeneous_confidential_computing_instance_public_IP> in the following command with the public IP address of the heterogeneous confidential computing instance that you created in Step 3: Create a heterogeneous confidential computing instance.

   • The following command sets the all_proxy='http://127.0.0.1:41000/' environment variable before the curl command is run. This ensures that requests sent by the curl command are encrypted by the Trusted Network Gateway client and sent through a secure channel.

   env all_proxy='http://127.0.0.1:41000/' \
       curl http://<heterogeneous_confidential_computing_instance_public_IP>:8080/v1/completions \
           -X POST \
           -H "Content-Type: application/json" \
           -d '{"model": "Qwen3-32B", "prompt": "Do you know the book Traction by Gino Wickman", "temperature": 0.0, "best_of": 1, "max_tokens": 132, "stream": false}'

   The following sample output indicates that the request sent by the client using the curl command is encrypted by the Trusted Network Gateway client and then sent through a secure channel. The large language model service that is deployed in the heterogeneous instance is successfully accessed.