Use CAA to deploy confidential containers on confidential VMs in ACK clusters - Container Service for Kubernetes

In scenarios where confidential computing is required, such as the financial risk control and healthcare industries, you can use Cloud API Adaptor (CAA), also known as Peer Pods, to deploy your workloads in a confidential computing environment that is built on top of Intel® Trust Domain Extensions (TDX). This helps you protect sensitive data from external attacks or potential risks in the offerings of cloud service providers and allows your workloads to meet industry compliance requirements.

Intel^® TDX is a CPU hardware-based security technology used by Elastic Compute Service (ECS) to secure confidential computing. For more information, see Introduction to TDX.

Feature overview

CAA is a key component of the Cloud Native Computing Foundation (CNCF) project Confidential Containers. CAA can integrate confidential computing with Kubernetes by calling API operations on the cloud platform to automatically create confidential VMs. The confidential VMs serve as sandboxes for workloads and provide a hardware-based trusted execution environment (TEE) to enhance runtime data security for workloads. This CAA-based solution provides the following benefits.

Ease of use: CAA eliminates the need to maintain underlying bare metal servers or nested virtualization stacks. After you deploy CAA in a Container Service for Kubernetes (ACK) cluster, you can use kubectl to deploy confidential workloads.
Security and confidentiality: CAA allows you to run workloads on ECS confidential VMs that are built on top of hardware-based memory encryption and isolation mechanisms to ensure runtime data integrity and confidentiality.
Compatibility with the community: ACK collaborates with the Confidential Containers community to support the deployment of confidential workloads. In addition, ACK enables code transparency to allow community auditing on a continuous basis.

You can deploy CAA as a DaemonSet in an ACK cluster. Then, you can deploy confidential workloads in the cluster. The confidential workloads run in pods that are deployed on confidential VMs. This ensures runtime data security for the workloads throughout the entire pod lifecycle and protects the workloads from external attacks. The following figure shows the architecture of the CAA-based solution.

Preparations

TDX confidential computing has certain limits. Make sure that you understand the limits before you use TDX confidential computing.

An ACK managed Pro cluster is created. The cluster must meet the following requirements. For more information, see Create an ACK managed cluster.

Parameter	Description
Region	Select China (Beijing).
Kubernetes Version	The Kubernetes version of the cluster must be 1.22 or later and earlier than 1.33. For more information about how to upgrade an ACK cluster, see Manually upgrade ACK clusters.
Network Plug-in	Select Flannel.
Configure SNAT for VPC	Select this check box. This feature enables the cluster to access the Internet. You can also enable this feature for an existing ACK cluster. For more information, see Enable an existing ACK cluster to access the Internet.
RRSA OIDC	Select Enable. The RAM Roles for Service Accounts (RRSA) feature allows you to limit the permissions to perform API operations on specific OSS volumes. This allows you to regulate access to cloud resources in a fine-grained manner and enhance cluster security. You can also enable this feature for an existing ACK cluster. For more information, see Use RRSA to authorize different pods to access different cloud services. After you enable this feature, refer to Obtain the URL and ARN information to obtain the Alibaba Cloud Resource Name (ARN) of the OpenID Connect (OIDC) provider for authentication.

You must select vSwitches in Zone I in the China (Beijing) region. For more information, see Create and manage vSwitches.

Step 1: Configure RRSA

To allow CAA to manage and configure confidential VMs, you must assign an identity (RAM role) and grant the required permissions (RAM policy) to CAA. This section describes how to authorize the service account used by Cloud API Adaptor to manage ECS instances and virtual private clouds (VPCs). This requires pod-specific permission control.

1. Create a RAM role

Refer to Create a RAM role for a trusted IdP to create a RAM role. In this example, the RAM role is named ack-caa-demo. The following table describes the key parameters of the RAM role.

Parameter	Description
Identity Provider Type	Select OIDC.
Identity Provider	Select ack-rrsa-<CLUSTER_ID>. <CLUSTER_ID> indicates the ID of your cluster.
Condition	oidc:iss: Use the default value. oidc:aud: Use the default value. oidc:sub: Manually add this condition to specify the service account that needs to assume the RAM role. In this example, the service account used by Cloud API Adaptor is in the confidential-containers-system namespace. Key: Select oidc:sub. Operator: Select StringEquals. Value: Enter `system:serviceaccount:confidential-containers-system:cloud-api-adaptor`.
Role Name	Set the role name to ack-caa-demo.

After you create the role, go to the role details page and record the ARN of the role in the Basic Information section for authentication.

2. Create a RAM policy

Use the following script to create an ack-caa-policy policy that provides the permissions to access ECS and VPC. For more information, see Create custom policies.

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:RunInstances",
        "ecs:DeleteInstance",
        "ecs:DescribeInstanceAttribute",
        "ecs:CreateNetworkInterface",
        "ecs:DeleteNetworkInterface",
        "ecs:AttachNetworkInterface",
        "ecs:ModifyNetworkInterfaceAttribute",
        "ecs:DescribeNetworkInterfaceAttribute"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "vpc:DescribeVSwitchAttributes",
        "vpc:AllocateEipAddress",
        "vpc:ReleaseEipAddress",
        "vpc:AssociateEipAddress",
        "vpc:UnassociateEipAddress",
        "vpc:DescribeEipAddresses"
      ],
      "Resource": "*"
    }
  ]
}

Attach the ack-caa-policy policy to the RAM role you created. For more information, see Grant permissions to a RAM role.

Step 2: Create a node pool and configure the security group rules of the node pool

Create a node pool that is used to deploy the CAA controller. Configure the security group rules of the node pool to open the ports used by CAA.

Log on to the ACK console. In the navigation pane on the left, click Clusters.
On the Clusters page, find the cluster to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

Click Create Node Pool and complete the node pool configurations by following the on-screen instructions.

The following table describes only the key parameters. For more information about the parameters of a node pool, see Create and manage a node pool.

Parameter	Description
Configure Managed Node Pool	Select Disable.
vSwitch	Select vSwitches in Zone I. Nodes in Zone I are added to the node pool.
Operating System	Select Ubuntu 22.04 64-bit.
Expected Nodes	The initial number of nodes in the node pool. You must specify at least one node.
Expand Advanced Options (Optional) to configure advanced settings.
Node Labels	Add the following labels to facilitate the scheduling of the CAA controller. Set the key to `node.kubernetes.io/worker`. Leave the value empty.

Go to the Node Pools page and click the name of the node pool you created. On the node pool details page, click the Overview tab. In the Node Pool Information section, click the security group ID to go to the security group details page.

On the Inbound tab, click Add Rule to add the following rules.

Protocol	Source	Destination	Description
Custom TCP	Current VPC CIDR Block	15150	Enables the CAA component to communicate with the confidential VM.
Custom UDP	Current VPC CIDR Block	4789	Enables network management based on Virtual Extensible Local Area Network (VXLAN).

Step 3: Deploy CAA

Deploy the following key components:

Confidential Containers (CoCo) Operator: This component automatically deploys and manages the runtime environment where confidential containers run. For example, the remote runtime of Kata Containers is Kata Remote Runtime.
CAA DaemonSet: CAA is deployed on nodes as a DaemonSet. This component dynamically creates confidential VMs as runtime nodes for pods based on the pod scheduling requirements in the cluster.

1. Install CoCo Operator

Run the following command to install CoCo Operator:

kubectl apply -k "https://github.com/AliyunContainerService/coco-operator/config/release"
kubectl apply -k "https://github.com/AliyunContainerService/coco-operator/config/samples/ccruntime/peer-pods"

2. Deploy CAA DaemonSet

Download the project source code.

git clone https://github.com/confidential-containers/cloud-api-adaptor.git -b v0.14.0
cd cloud-api-adaptor

Modify the security group ID and vSwitch IDs specified in the src/cloud-api-adaptor/install/overlays/alibabacloud/kustomization.yaml file to connect the VM to the network environment.

Parameter	Description
`SECURITY_GROUP_IDS`	Set the value to the ID of the security group of the node pool created in Step 2.
`VSWITCH_ID`	Set the value to the IDs of the vSwitches (in Zone I) used by the node pool created in Step 2.

Create a src/cloud-api-adaptor/install/overlays/alibabacloud/alibabacloud-cred.env file to configure authentication information. The following code shows the file content:
Replace <role_arn> with the ARN of the RAM role and <provider_arn> with the ARN of the RRSA OIDC provider.
```
ALIBABA_CLOUD_ROLE_ARN=<role_arn>
ALIBABA_CLOUD_OIDC_PROVIDER_ARN=<provider_arn>
ALIBABA_CLOUD_OIDC_TOKEN_FILE=/var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
```

Deploy the CAA workloads.

kubectl apply -k src/cloud-api-adaptor/install/overlays/alibabacloud

Wait 3 minutes and check the deployment progress.

kubectl -n confidential-containers-system get pod

Expected output:

NAME                                              READY   STATUS    RESTARTS   AGE
cc-operator-controller-manager-5d79465b47-d8s2k   1/1     Running   0          2m11s
cc-operator-daemon-install-trlvt                  1/1     Running   0          108s
cc-operator-pre-install-daemon-qpvzw              1/1     Running   0          117s
cloud-api-adaptor-daemonset-46spp                 1/1     Running   0          92s

If cc-operator-* and cloud-api-adaptor-daemonset-* are displayed in the output and they are in the Running state, the CAA workloads are deployed.

Step 4: Deploy an application

Perform the following operations to deploy an application. In the application configurations, the runtimeClassName: kata-remote setting specifies the confidential container runtime. When pods are scheduled, Kata Remote triggers CAA to dynamically create TDX confidential VMs.

Important

Before deleting the cluster, ensure all workloads with runtimeClassName: kata-remote are removed to prevent residual confidential VM resources created by CAA.

Use the following YAML template to create a pod-caa-demo.yaml file, which is used to create a pod that uses the confidential container runtime.

apiVersion: v1
kind: Pod
metadata:
  name: pod-caa-demo
spec:
  runtimeClassName: kata-remote
  containers:
    - image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
      name: hello
      command:
      - sh
      - -c
      - 'echo hello && sleep infinity'

Deploy the pod-caa-demo.yaml file.

kubectl apply -f pod-caa-demo.yaml

Wait 3 minutes and check the deployment progress.

kubectl get pod pod-caa-demo

Expected output:

NAME           READY   STATUS    RESTARTS   AGE
pod-caa-demo   1/1     Running   0          52s

Log on to the ECS console. In the left-side navigation pane, click Instances. If a TDX confidential VM whose name starts with podvm- is displayed, CAA has created underlying computing resources for the application.

References

Contact us

If you have any questions or suggestions for ACK, join the DingTalk group 30521601 to contact us.