Container Service for Kubernetes:Secure confidential container environments with remote attestation

Use case 1: Pre-launch environment verification

This process is designed for cluster administrators to ensure that pods are only launched in authentic and trusted confidential environments. The entire workflow is automated and requires no manual intervention.

1. Initiate deployment: An administrator creates a workload by submitting a pod manifest to the cluster's API server. The kubelet on the node then requests the creation of a pod sandbox from containerd via the Container Runtime Interface (CRI). containerd checks the pod's runtimeClassName and invokes the underlying Kata Remote runtime kata-runtime.

2. Create confidential VM: Kata Remote responds to the request by calling the Alibaba Cloud OpenAPI to launch a lightweight and hardware-isolated ECS TDX confidential VM, which serves as the Kubernetes pod.

3. Mandatory remote attestation: Before the image is pulled, the Attestation Agent inside the VM automatically generates evidence of the runtime environment, including hardware signatures, and send it to the Trustee service for verification.

4. Validation and workload execution: Only after the Trustee confirms that the evidence is valid does the guest's control logic proceed with subsequent operations, including pulling the application image and starting the container.

Use case 2: Runtime remote attestation

This capability is for running containerized applications, allowing them to proactively prove the security of their own environment to external services, thereby building a chain of trust across systems.

1. Application fetch evidence: The application inside the container requests a real-time remote attestation report by calling a local API operation at http://127.0.0.1:8006/aa/evidence.

2. External service verification: The application sends this evidence to an external service (the consumer). The service consumer then forwards the evidence to its own trusted Trustee service for validation. A successful validation proves that the communicating party is running in a protected confidential environment, thus establishing trust dynamically.

1. Configure the Trustee

On the Trustee server, configure policies to define which environment characteristics are considered trustworthy.

1. Configure the container security launch policy.

   This policy file is treated as a confidential resource and is read during the TDX pod's startup process to trigger the remote attestation flow.

   # Create a directory to store policies
   mkdir -p /opt/trustee/kbs/repository/default/container-policy
   
   # Write a policy file that allows all images (for demo purposes only)
   cat << EOF > /opt/trustee/kbs/repository/default/container-policy/insecure-accept-all
   {
       "default": [{"type": "insecureAcceptAnything"}],
       "transports": {}
   }
   EOF

2. Configure the remote attestation policy.

   This policy defines the specific standards for trusted hardware and software environments. For example, it can require the environment to run on the Intel TDX platform.

   # Modify the policy file /opt/trustee/attestation-service/policies/opa/default.rego
   # Replace the content with the remote attestation policy for ECS TDX PeerPods
   cat << EOF > /opt/trustee/attestation-service/policies/opa/default.rego
   package policy
   
   import rego.v1
   
   default executables := 32
   default hardware := 97
   default configuration := 32
   default file_system := 32
   
   ##### ECS TDX PeerPod
   executables := 3 if {
   	input.tdx
   }
   
   hardware := 2 if {
   	# Verify that the evidence is of the Intel TDX type
   	input.tdx.quote.header.tee_type == "81000000"
   	# Verify the vendor ID
   	input.tdx.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"
   }
   
   configuration := 2 if {
   	input.tdx
   }
   
   file_system := 2 if {
   	input.tdx
   }
   EOF

3. Restart the Trustee service to apply the policy.

   service as restart

2. Deploy a pod with remote attestation enabled

1. Prepare the initdata and deploy the pod.

   initdata is used to pass configuration information to the PeerPod, including the Trustee service address.

   # Replace with the actual Trustee IP address
   TRUSTEE_SERVICE_ENDPOINT=1.2.3.4
   
   # Generate the initdata.toml file
   cat <<EOF > initdata.toml
   version = "0.1.0"
   algorithm = "sha256"
   [data]
   "aa.toml" = '''
   [eventlog_config]
   enable_eventlog = true
   [token_configs]
   [token_configs.kbs]
   url = "http://${TRUSTEE_SERVICE_ENDPOINT}:8080"
   '''
   "cdh.toml" = '''
   [kbc]
   name = "cc_kbc"
   url = "http://${TRUSTEE_SERVICE_ENDPOINT}:8080"
   [image]
   image_security_policy_uri = "kbs:///default/container-policy/insecure-accept-all"
   '''
   EOF
   
   # Compress the initdata with gzip and encode it with Base64
   initdata=$(cat initdata.toml | gzip | base64 -w0)

   Run echo $initdata to view the encoded content of initdata.

2. Create the pod YAML file.

   Write the initdata string generated in the previous step to the annotations field of the pod's metadata.

   # Create a file named pod.yaml
   cat << EOF > pod.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-caa-demo
     annotations:
       io.katacontainers.config.hypervisor.cc_init_data: ${initdata}
   spec:
     runtimeClassName: kata-remote
     containers:
       - image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
         name: hello
         command: ["sh", "-c", "echo hello && sleep infinity"]
   EOF

3. Deploy the pod.

   kubectl apply -f pod.yaml

   After deployment, run kubectl get po pod-caa-demo to check the pod status.

4. Verify the remote attestation status on the Trustee server.

   Check the logs of the remote attestation service to confirm that the environment validation was successful.

   journalctl -u as -f 

   If you see the message Tdx Verifier/endorsement check passed in the logs, it indicates that the remote attestation was successful and the pod has started in a trusted environment.

1. Get remote attestation evidence from within the container

curl 127.0.0.1:8006/aa/evidence?runtime_data=AAAA

2. Prepare the verification environment

The verification process relies on an independent and trusted Trustee instance.

1. (Optional) Run yum install trustee -y to install the Trustee on your verifier server.

2. Configure the client-side validation policy.

   On the verifier node, create a new policy file /opt/trustee/attestation-service/policies/opa/client.rego in the Trustee service.

   # Create the client.rego file with the following content
   cat <<EOF > /opt/trustee/attestation-service/policies/opa/client.rego
   
   # Policy to verify the evidence obtained from the confidential container (TEE)
   package policy
   import rego.v1
   
   # --- Default scores ---
   default executables := 32
   default hardware := 97
   default configuration := 32
   default file_system := 32
   
   # --- Validation rules for Alibaba Cloud ECS TDX PeerPods ---
   # For example: score of 2 = "trustworthy", 3 = "affirming"
   # If the input evidence contains a "tdx" field, the software stack is considered "affirming"
   executables := 3 if {
       input.tdx
   }
   
   # Hardware scoring rule: If both of the following conditions are met, the hardware environment is rated as "trustworthy" (score of 2).
   hardware := 2 if {
       # Check the TEE type field. It must be the Intel TDX identifier "81000000"
       input.tdx.quote.header.tee_type == "81000000"
   
       # Check the vendor ID field. It must be the official Intel UUID
       input.tdx.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"
   }
   
   # If the input is TDX evidence, its configuration is considered "trustworthy".
   configuration := 2 if {
       input.tdx
   }
   
   # If the input is TDX evidence, its file system is considered "trustworthy".
   file_system := 2 if {
       input.tdx
   }
   EOF

3. Restart the Trustee to apply the client.rego policy.

   service as-restful restart

3. Prepare and send the verification request

Submit the evidence obtained from the container to the verification service.

1. (Recommended) Securely store the original evidence.

   To avoid errors from special characters, we recommend that you save the evidence to a file.

   # Create a temporary file named evidence.json
   cat > evidence.json

   After running the command, paste the complete JSON string. Then, save the file and exit edit mode.

2. Read the evidence from the file and prepare the request.

   # Read the content from the file into a variable
   EVIDENCE=$(cat evidence.json)
   
   # Base64-encode the evidence and make it URL-safe
   BASE64EVIDENCE=$(echo ${EVIDENCE} | base64 -w 0 2>/dev/null | tr '+/' '-_' | tr -d '=' )
   
   # Create a request body file that contains the encoded evidence
   cat << EOF > request.json
   {
       "verification_requests": [
           {
               "tee": "tdx",
               "evidence": "${BASE64EVIDENCE}"
           }
       ],
       "policy_ids": ["client"]
   }
   EOF

3. Send the verification request.
   Send a POST request to the /attestation endpoint of your local verification service on the default port 50005.

   curl -k -X POST http://127.0.0.1:50005/attestation \     -i \     -H 'Content-Type: application/json' \     -d @request.json

   The response body will contain a token field, which is a JWT string. This is your verification token. Copy the token.

4. Parse and evaluate the verification result

Parse the JWT token to retrieve the trustworthiness scores for each sub-state of the current environment.

1. Store the verification token.

   # Replace '<...>' with the actual JWT string you obtained (without the quotes)
   TOKEN='<Paste the JWT token here>'

2. Parse the token and view the scores.

   echo $TOKEN | cut -d'.' -f2 | base64 -d |  jq '.submods.cpu0."ear.trustworthiness-vector"'

   Expected output:

   base64: invalid input
   {
     "configuration": 2,
     "executables": 3,
     "file-system": 2,
     "hardware": 2
   }

   • The base64: invalid input message can be safely ignored.

   • The JSON content shows the verification result. The scores match the values defined in your policy file, confirming that the container is running in a compliant and trusted TDX environment. The verification is successful.