Platform For AI:Secure encryption inference service

Overview

Take the following steps for encrypted deployment and inference:

1. Step 1: Prepare an encrypted model

   • Encrypt the model: Use Gocryptfs or Sam to encrypt the model and upload the encrypted model to a data source (such as Object Storage Service (OSS)) for subsequent mounting and deployment in EAS.

   • Store decryption keys: Currently, EAS supports the self-built Trustee remote attestation service hosted by Alibaba Cloud. Deploy the Trustee remote attestation service in Alibaba Cloud Container Service for Kubernetes (ACK) Serverless to verify the deployment and inference environment, and use Key Management Service (KMS) as the storage backend for decryption keys.

2. Step 2: Deploy the encrypted model in PAI-EAS

   When you deploy encrypted models using PAI-EAS, the system automatically connects to the Trustee remote attestation service to verify the deployment environment. After verification, the system uses the key in KMS to decrypt and mount the model, and then deploys it as an EAS service.

3. Step 3: Call the service for secured inference

   After deployment, users can access the inference service to initiate inference requests.

Step 1: Prepare an encrypted model

Before deploying a model, you need to encrypt the model and upload it to cloud storage. The key for decrypting the model will be hosted by KMS, controlled by the remote attestation service. Perform model encryption operations in a local or trusted environment. This solution shows how to deploy Qwen2.5-3B-Instruct as an example.

1. Prepare a model (optional)

Run the following command in the terminal use ModelScope tool to download Qwen2.5-3B-Instruct(requires Python 3.9 or higher).

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

The command will download the model to ~/.cache/modelscope/hub/models/Qwen/Qwen2.5-3B-Instruct/.

2. Encrypt the model

PAI-EAS supports the following encryption methods. This solution uses Sam as an example.

• Gocryptfs: An encryption mode based on AES256-GCM that complies with the open-source Gocryptfs standard.

• Sam: Alibaba Cloud's trusted AI model encryption format that protects model confidentiality and prevents License content from being tampered with or illegally used.

3. Upload the model

EAS supports multiple storage services for the encrypted model and can load the decrypted model into service instance through mounting during deployment. For more information, see Mount storage.

Taking Object Storage Service (OSS) as an example, refer to the Get started with the OSS console to create a storage space (Bucket) and a directory named qwen-encrypted (for example, oss://examplebucket/qwen-encrypted/). Due to the large size of model files, it is recommended to use ossbrowser to upload the encrypted model to this directory.

The following screenshot shows the uploaded model using Sam encryption. If you use Gocryptfs encryption, the file names will be encrypted as garbled text.

4. Set up remote attestation service and upload the key

Host the key in the remote attestation service. The remote attestation service is a user-managed verification service responsible for verifying the running environment of the model and inference service. When the trustworthiness of the EAS deployment environment meets the expected conditions, the key will be injected to decrypt and mount the model.

Currently, EAS supports the self-built Trustee remote attestation service hosted by Alibaba Cloud. You can use ACK Serverless to deploy Trustee to verify the model deployment and inference environment. Meanwhile, you can use KMS to provide professional protection for the model decryption key. Perform the following steps:

1. First, create a KMS instance to store the key.

   1. Go to the KMS console, choose Resource > Instances in the left-side navigation pane. Then, create and start an instance on the Software Key Management tab. When starting the instance, select the same VPC as the ACK cluster. For more information, see Purchase and enable a KMS instance.

      Wait for about 10 minutes for the instance to start.

   2. After the instance is started, choose Resource > Keys in the left-side navigation pane. Then, create a customer master key for the instance on the Keys page. For more information, see Step 1: Create a software-protected key.

   3. Choose Application Access > AAPs in the left-side navigation pane. Then, create an application access point (AAP) for the instance on the AAPs page. Set Scope to the created KMS instance. For more information, see Mode 1: Quick creation.

      After the AAP is created, your browser automatically downloads a ClientKey***.zip file. This zip file contains the following files after extraction:

      • ClientKeyContent: The default file name is clientKey_****.json.

      • ClientKeyPassword: The default file name is clientKey_****_Password.txt.

   4. Go to Resource > Instances, click the KMS instance name. Then, in the Basic Information section, click Download next to Instance CA Certificate to export the public key certificate file PrivateKmsCA_***.pem of the KMS instance.

2. Create an ACK service cluster and install the csi-provisioner component.

   1. Go to the Create Cluster page to create an ACK Serverless cluster. Take note of the following parameters. For more parameter settings, see Create an ACK Serverless cluster.

      1. Cluster Configurations: Configure the following parameters and click Next: Component Configurations.

         Key item	Description
         VPC	Select Use Existing and select Configure SNAT For VPC. Otherwise, you cannot pull the Trustee image.
         vSwitch	Make sure that at least two vSwitches are created in the existing VPC. Otherwise, you cannot expose the public ALB.

      2. Component Configurations: Configure the following parameters and click Next: Confirm.

         Key item	Description
         Service Discovery	Select CoreNDS.
         Ingress	Select ALB Ingress. For Gateway Source, select New and select two vSwitches.

      3. Confirm: Confirm the configuration information and terms of service, and click Create Cluster.

   2. After the cluster is created, install the csi-provisioner (Managed) component. For more information, see Manage components.

3. Deploy the Trustee in the ACK cluster.

   1. First, connect to the cluster over the Internet or an internal network. For more information, see Connect to a cluster.

   2. Upload the downloaded client key content (clientKey_****.json), key password (clientKey_****_Password.txt), and CA certificate (PrivateKmsCA_***.pem) to the environment connected to the ACK Serverless cluster, and run the following command to deploy Trustee using Alibaba Cloud KMS as the key storage backend.

      # Install the plugin
      helm plugin install https://github.com/AliyunContainerService/helm-acr
      
      helm repo add trustee acr://trustee-chart.cn-hangzhou.cr.aliyuncs.com/trustee/trustee
      helm repo update
      
      export DEPLOY_RELEASE_NAME=trustee
      export DEPLOY_NAMESPACE=default
      export TRUSTEE_CHART_VERSION=1.0.0
      
      # Set the region information of the ACK cluster, such as cn-hangzhou
      export REGION_ID=cn-hangzhou
      
      # KMS instance information exported earlier 
      # Replace with your KMS instance ID
      export KMS_INSTANCE_ID=kst-hzz66a0*******e16pckc
      # Replace with the path to your KMS instance application identity secret 
      export KMS_CLIENT_KEY_FILE=/path/to/clientKey_KAAP.***.json
      # Replace with the path to your KMS instance secret security token 
      export KMS_PASSWORD_FILE=/path/to/clientKey_KAAP.***_Password.txt
      # Replace with the path to your KMS instance CA certificate
      export KMS_CERT_FILE=/path/to/PrivateKmsCA_kst-***.pem
      
      helm install ${DEPLOY_RELEASE_NAME} trustee/trustee \
        --version ${TRUSTEE_CHART_VERSION} \
        --set regionId=${REGION_ID} \
        --set kbs.aliyunKms.enabled=true \
        --set kbs.aliyunKms.kmsIntanceId=${KMS_INSTANCE_ID} \
        --set-file kbs.aliyunKms.clientKey=${KMS_CLIENT_KEY_FILE} \
        --set-file kbs.aliyunKms.password=${KMS_PASSWORD_FILE} \
        --set-file kbs.aliyunKms.certPem=${KMS_CERT_FILE} \
        --namespace ${DEPLOY_NAMESPACE}

      Note

      The first command (helm plugin install...) may take a long time to execute. If the installation fails, you can first uninstall the plugin by running helm plugin uninstall cm-push, and then run the plugin installation command again.

      Sample output:

      NAME: trustee
      LAST DEPLOYED: Tue Feb 25 18:55:33 2025
      NAMESPACE: default
      STATUS: deployed
      REVISION: 1
      TEST SUITE: None

   3. Run the following command in the environment connected to the ACK Serverless cluster to obtain the access address of Trustee.

      export TRUSTEE_URL=http://$(kubectl get AlbConfig alb -o jsonpath='{.status.loadBalancer.dnsname}')/${DEPLOY_RELEASE_NAME}
      echo ${TRUSTEE_URL}

      Sample output: http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/trustee.

   4. Run the following command in the environment connected to the ACK Serverless cluster to test the connectivity of the Trustee service.

      cat << EOF | curl -k -X POST ${TRUSTEE_URL}/kbs/v0/auth -H 'Content-Type: application/json' -d @-
      {
          "version":"0.1.0",
          "tee": "tdx",
          "extra-params": "foo"
      }
      EOF

      If Trustee is running normally, the expected output is:

      {"nonce":"PIDUjUxQdBMIXz***********IEysXFfUKgSwk=","extra-params":""}

4. Configure the Trustee network whitelist.

   Note

   The purpose is to allow the PAI-EAS deployment environment to actively access the remote attestation service for security checks.

   1. Go to the ALB console, create an access control list (ACL), and add the address/address segment with Trustee access permissions as IP entries. For more information, see Access control. Address segments to be added:

      • The public IP address of the VPC bound with the EAS service.

      • The egress IP address of the inference client.

   2. Run the following command to obtain the ALB instance ID used by the trustee instance in the cluster:

      kubectl get ing --namespace ${DEPLOY_NAMESPACE} kbs-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | cut -d'.' -f1 | sed 's/[^a-zA-Z0-9-]//g'

      Expected output:

      alb-llcdzbw0qivhk0****

   3. In the left-side navigation pane of the ALB console, choose ALB > Instances. In the region where the cluster is located, search for the ALB instance obtained in the previous step. Click the instance ID to go to the instance details page. Then, in the Instance Information section at the bottom of the page, click Disable Configuration Read-only Mode.

   4. Switch to the Listeners tab, click Enable in the Access Control column of the target listener instance, and configure the whitelist as the ACL created in the previous step.

5. Create a secret to store the model decryption key.

   The model decryption key hosted by the trustee is stored in KMS. The key can be accessed only after the remote attestation service verifies the destination environment.

   Go to the KMS console, choose Resource > Secrets in the left-side navigation pane. Then, choose Generic Secrets and click Create a secret. Take note of the following parameters:

   • Secret Name: A custom secret name used to index the key, such as model-decryption-key.

   • Secret Value: Enter the key used to encrypt the model. For example, 0Bn4Q1wwY9fN3P. Use your actual key.

   • CMK: Select the master key created in the previous step.

Step 2: Deploy the encrypted model in PAI-EAS

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).

2. On the Elastic Algorithm Service (EAS) page, click Deploy Service. Then, click Custom Deployment in the Custom Model Deployment section.

3. On the Custom Deployment page, configure the following key parameters. For more parameter information, see Custom deployment.

   Parameter		Description
   Environment Information	Deployment Method	Select Image-based Deployment.
   	Image Configuration	Select an image. In this solution, select Alibaba Cloud Image > chat-llm-webui:3.0-vllm.
   	Mount storage	Click + OSS and configure the following parameters: Uri: Select the directory where the encrypted model is located. For example, oss://examplebucket/qwen-encrypted/. Use your actual path. Mount Path: Configure the directory where the plaintext model is mounted, such as /mnt/model.
   	Command	Example configuration: python webui/webui_server.py --port=8000 --model-path=/mnt/model --backend=vllm. Note that --model-path must be the same as the mount path to read the decrypted model.
   	Port Number	In this solution, set to 8000.
   	Environment Variables	You can add environment variables for Trustee verification. In this solution, set: key eas-test: value 123.
   Resource Information	Deployment Resources	In this solution, use ml.gu7i.c32m188.1-gu30.
   VPC	VPC	Configure a VPC and set the SNAT public network egress IP for the VPC to enable public network access for EAS, so that it can access the remote attestation service and perform environment security checks.
   	vSwitch
   	Security Group Name
   Features	Configure Secure Encryption Environment	Turn on the Configure Secure Encryption Environment switch and configure the following parameters: Encryption Method: The encryption method used to encrypt the model. Supports Sam and Gocryptfs. This solution selects Sam. System Trust Management Service Address: The address of the deployed Trustee service. For example, http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/trustee. KBS URI of Decryption Key: The KBS URI of the model decryption key used, in the format kbs:///default/aliyun/<secret_name_of_the_key>. Replace <secret_name_of_the_key> with the secret name created in the previous step.

   The final JSON configuration example is as follows:

   Click here to view the JSON configuration

   {
       "cloud": {
           "computing": {
               "instances": [
                   {
                       "type": "ml.gu7i.c32m188.1-gu30"
                   }
               ]
           },
           "networking": {
               "security_group_id": "sg-2vcbmhs3puagy23k****",
               "vpc_id": "vpc-2vctxcm4qncgriz0j****",
               "vswitch_id": "vsw-2vc0tdylfux849zb7****"
           }
       },
       "confidential": {
           "decryption_key": "kbs:///default/aliyun/model-decryption-key",
           "trustee_endpoint": "http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/trustee"
       },
       "containers": [
           {
               "env": [
                   {
                       "name": "eas-test",
                       "value": "1234"
                   }
               ],
               "image": "eas-registry-vpc.cn-hangzhou.cr.aliyuncs.com/pai-eas/chat-llm-webui:3.0.5-vllm",
               "port": 8000,
               "script": "python webui/webui_server.py --port=8000 --model-path=--model-path=/mnt/data/model --backend=vllm"
           }
       ],
       "metadata": {
           "cpu": 8,
           "gpu": 1,
           "instance": 1,
           "memory": 30000,
           "name": "xynnn_eas_test_gpu"
       },
       "storage": [
           {
               "encryption": {
                   "method": "sam"
               },
               "mount_path": "/mnt/data/model",
               "oss": {
                   "path": "oss://examplebucket/qwen-encrypted/"
               },
               "properties": {
                   "resource_type": "model"
               }
           }
       ]
   }

4. Click Deploy.

   When the Service Status is Running, the service has been successfully deployed. At this point, go to the Clusters page, click the cluster name, and then on the Pods page, follow the operation guide in the screenshot below to view the attestation-service-* container logs. The logs will show whether Trustee has verified the model deployment environment and display detailed execution environment information.

Step 3: Call the service for secured inference

1. View the service endpoint

On the Model Online Service (EAS) page, click Invocation Method in the Service Type column of the target service to obtain the service endpoint and token.

2. Call the service

1. You can run the following cURL command to send an inference request to the service.

curl <Service_URL> \
  -H "Content-type: application/json" \
  --data-binary @openai_chat_body.json \
  -v \
  -H "Connection: close" \
  -H "Authorization: <Token>"

Where:

• <Service_URL>: The EAS service endpoint.

• <Token>: The EAS service token.

• openai_chat_body.json is the original inference request. An example of the request content is as follows:

  {
      "max_new_tokens": 4096,
      "use_stream_chat": false,
      "prompt": "What is the capital of Canada?",
      "system_prompt": "Act like you are a knowledgeable assistant who can provide information on geography and related topics.",
      "history": [
          [
              "Can you tell me what's the capital of France?",
              "The capital of France is Paris."
          ]
      ],
      "temperature": 0.8,
      "top_k": 10,
      "top_p": 0.8,
      "do_sample": true,
      "use_cache": true
  }

The following response is returned for the preceding example:

{
    "response": "The capital of Canada is Ottawa.",
    "history": [
        [
            "Can you tell me what's the capital of France?",
            "The capital of France is Paris."
        ],
        [
            "What is the capital of Canada?",
            "The capital of Canada is Ottawa."
        ]
    ]
}