Platform For AI:Secure encrypted inference service

1. Prepare a model (optional)

The Qwen2.5-3B-Instruct model requires Python 3.9 or later. To download the model using the ModelScope tool, run the following command in the terminal.

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

After success, the model downloads to ~/.cache/modelscope/hub/models/Qwen/Qwen2.5-3B-Instruct/.

2. Encrypt the model

PAI-EAS supports two encryption methods. This example uses Sam.

• Gocryptfs: An encryption mode that is based on AES-256-GCM and complies with the open source Gocryptfs standard.

• Sam: An Alibaba Cloud trusted AI model encryption format that protects model confidentiality and prevents license content from being tampered with or used illegally.

3. Upload the model

EAS supports various storage backends for encrypted models. Mount decrypted models into service instances during deployment. For more information, see Mount storage.

This example uses OSS. Create a bucket and directory (e.g., oss://examplebucket/qwen-encrypted/). See Quick Start. For large model files, use ossbrowser to upload.

The following figure shows the result after uploading the Sam-encrypted model. If you use Gocryptfs, filenames appear as encrypted garbled characters.

4. Set up a remote attestation service and upload the key

The decryption key is managed by the remote attestation service, which verifies the runtime environment of the model and inference service. The key is injected only when the EAS environment meets the expected trust conditions.

Deploy the Trustee remote attestation service in an ACK serverless cluster. Use Alibaba Cloud KMS for secure key storage.

1. Create an Alibaba Cloud KMS instance as the key storage backend.

   1. Go to the Key Management Service console. In the left pane, choose Resource > Instance Management. On the Software Key Management tab, create and start an instance. When you start the instance, select the same VPC as the ACK cluster. For more information, see Purchase and enable a KMS instance.

      Wait about 10 minutes for the instance to start.

   2. After the instance starts, in the left pane, choose Resource > Key Management. On the Key Management page, create a customer master key (CMK) for the instance. For more information, see Step 1: Create a software-protected key.

   3. In the left pane, choose Application Access > Access Point. On the Access Point page, create an application access point for the instance. Set Scope to the created KMS instance. For other parameters, see Method 1: Quick creation.

      After the application access point is created, the browser automatically downloads a ClientKey***.zip file. After you decompress the .zip file, it contains the following files:

      • Client Key content (ClientKeyContent): The default filename is clientKey_****.json.

      • Credential password (ClientKeyPassword): The default filename is clientKey_****_Password.txt.

   4. On the Resource > Instance Management page, click the name of the KMS instance. In the Basic Information section, click Download next to Instance CA Certificate to export the public key certificate file PrivateKmsCA_***.pem of the KMS instance.

2. You can create an ACK cluster and install the csi-provisioner component.

   1. Go to the Create Cluster page to create an ACK serverless cluster. The following table describes the key parameters. For other parameters, see Create an ACK cluster.

      1. Cluster Configurations: Configure the following parameters. Then, click Next: Component Configurations.

         Key configuration	Description
         VPC	Select Use Existing and select Configure SNAT For VPC. Otherwise, you cannot pull the Trustee image.
         VSwitch	Make sure that at least two virtual switches are created in the existing VPC. Otherwise, you cannot expose the public ALB Ingress.

      2. Component Configurations: Configure the following parameters. Then, click Next: Confirm Configurations.

         Key configuration	Description
         Service Discovery	Select CoreDNS.
         Ingress	Select ALB Ingress. For the source of the ALB cloud-native gateway instance, select New and select two virtual switches.

      3. Confirm Configurations: Confirm the configuration information and terms of service. Then, click Create Cluster.

   2. After the cluster is created, install the csi-provisioner (Managed) component. For more information, see Manage components.

3. Deploy the Trustee remote attestation service in the ACK cluster.

   1. Connect to the cluster over the Internet or an internal network. For more information, see Connect to a cluster.

   2. Upload the downloaded KMS instance application identity credential (clientKey_****.json), credential password (clientKey_****_Password.txt), and CA certificate (PrivateKmsCA_***.pem) to the environment that is connected to the ACK serverless cluster. Run the following command to deploy the Trustee remote attestation service and use Alibaba Cloud KMS as the key storage backend.

      # Install the plugin.
      helm plugin install https://github.com/AliyunContainerService/helm-acr
      
      helm repo add trustee acr://trustee-chart.cn-hangzhou.cr.aliyuncs.com/trustee/trustee
      helm repo update
      
      export DEPLOY_RELEASE_NAME=trustee
      export DEPLOY_NAMESPACE=default
      export TRUSTEE_CHART_VERSION=1.7.6
      
      # Set the region of the ACK cluster, for example, cn-hangzhou.
      export REGION_ID=cn-hangzhou
      
      # Information about the exported KMS instance. 
      # Replace with your KMS instance ID.
      export KMS_INSTANCE_ID=kst-hzz66a0*******e16pckc
      # Replace with the path to your KMS instance application identity credential. 
      export KMS_CLIENT_KEY_FILE=/path/to/clientKey_KAAP.***.json
      # Replace with the path to your KMS instance credential password. 
      export KMS_PASSWORD_FILE=/path/to/clientKey_KAAP.***_Password.txt
      # Replace with the path to your KMS instance CA certificate.
      export KMS_CERT_FILE=/path/to/PrivateKmsCA_kst-***.pem
      
      helm install ${DEPLOY_RELEASE_NAME} trustee/trustee \
        --version ${TRUSTEE_CHART_VERSION} \
        --set regionId=${REGION_ID} \
        --set kbs.aliyunKms.enabled=true \
        --set kbs.aliyunKms.kmsIntanceId=${KMS_INSTANCE_ID} \
        --set-file kbs.aliyunKms.clientKey=${KMS_CLIENT_KEY_FILE} \
        --set-file kbs.aliyunKms.password=${KMS_PASSWORD_FILE} \
        --set-file kbs.aliyunKms.certPem=${KMS_CERT_FILE} \
        --namespace ${DEPLOY_NAMESPACE}

      Note

      The first command to install the plugin (helm plugin install...) may take a long time to run. If the installation fails, you can run the helm plugin uninstall cm-push command to uninstall the plugin and then run the plugin installation command again.

      The following example shows a sample output:

      NAME: trustee
      LAST DEPLOYED: Tue Feb 25 18:55:33 2025
      NAMESPACE: default
      STATUS: deployed
      REVISION: 1
      TEST SUITE: None

   3. In the environment that is connected to the ACK serverless cluster, run the following command to obtain the endpoint of Trustee.

      export TRUSTEE_URL=http://$(kubectl get AlbConfig alb-$DEPLOY_RELEASE_NAME -o jsonpath='{.status.loadBalancer.dnsname}')/api
      echo ${TRUSTEE_URL}

      A sample output is http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/api.

   4. In the environment that is connected to the ACK serverless cluster, run the following command to test the connectivity of the Trustee service.

      cat << EOF | curl -k -X POST ${TRUSTEE_URL}/kbs/v0/auth -H 'Content-Type: application/json' -d @-
      {
          "version":"0.4.0",
          "tee": "tdx",
          "extra-params": "foo"
      }
      EOF

      If the Trustee service is running normally, the expected output is as follows:

      {"nonce":"PIDUjUxQdBMIXz***********IEysXFfUKgSwk=","extra-params":""}

4. Configure a network whitelist for Trustee.

   Note

   This configuration allows the PAI-EAS model deployment environment to access the remote attestation service for environment security checks.

   1. Go to the Alibaba Cloud Application Load Balancer (ALB) console, create an access control policy group, and add the IP addresses or CIDR blocks that have permissions to access Trustee as IP entries. For more information, see Access control. The CIDR blocks that need to be added are as follows:

      • The public IP address of the VPC to which the EAS service is bound during deployment.

      • The egress IP address of the inference client.

   2. Run the following command to obtain the ID of the Server Load Balancer (SLB) instance that is used by the Trustee instance on the cluster.

      kubectl get ing --namespace ${DEPLOY_NAMESPACE} frontend-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | cut -d'.' -f1 | sed 's/[^a-zA-Z0-9-]//g'

      The expected output is as follows:

      alb-llcdzbw0qivhk0****

   3. In the navigation pane on the left of the Alibaba Cloud ALB console, choose Application Load Balancer > Instances. In the region where the cluster is located, search for the ALB instance that you obtained in the preceding step and click the instance ID to go to the instance details page. At the bottom of the page, in the Instance Attributes section, click Disable Configuration Read-only Mode.

   4. Switch to the Listeners tab. In the access control column of the target listener, click **Enable** and configure the whitelist as the access control policy group that you created in the preceding step.

5. Create a secret to store the model decryption key.

   The model decryption key managed by Trustee is stored in KMS. The key can be accessed only after the remote attestation service verifies the target environment.

   Go to the Key Management Service console. In the left pane, choose Resource > Secrets Manager. On the Generic Secret tab, click Create Secret. The following table describes the key parameters.

   • Secret Name: A custom secret name that is used to index the key. Example: model-decryption-key.

   • Set Secret Value: The key that is used to encrypt the model. Example: `0Bn4Q1wwY9fN3P`. Use your actual key.

   • Encryption Master Key: Select the master key that you created in the preceding step.

6. Log on to the Trustee management interface to view historical key access records.

   1. Obtain the address of the Trustee frontend management interface.

      kubectl get AlbConfig alb-$DEPLOY_RELEASE_NAME -o jsonpath='{.status.loadBalancer.dnsname}'

      Access the address in a browser.

   2. Obtain the private key for logging on to the Trustee frontend management interface.

      kubectl get secret kbs-auth-keypair -o jsonpath="{.data.private\.key}" | base64 -d

      Paste the key into the frontend interface to complete the logon.

   3. On the Trustee Gateway Management Platform, click Audit Log in the navigation pane on the left. Click Resource Audit Interface in the middle of the page to view historical records of model decryption key access.