Elastic Compute Service:Deploy CoCo on Alibaba Cloud bare metal

Step 1: Create a TDX bare metal instance

Create an 8th-generation Alibaba Cloud bare metal instance that supports Intel TDX technology.

1. Go to the instance purchase page.

2. Select a billing method, region, and availability zone. The required community image is only available in specific availability zones. For this example, select Beijing Zone I.

3. In the Instance configuration section, click Elastic Bare Metal Server, and then select the ecs.ebmg8i.48xlarge instance type.

4. In the Image configuration section, select Community Image and search for the image using the image ID m-2ze2ucup4c5bvgx751lx. Below the image selection, you must select the Confidential VM checkbox to enable the TDX feature.

   Important

   This is a custom Ubuntu-based image with pre-installed drivers that support TDX. The default SSH login user is ubuntu, not root.

5. Configure settings for the network, storage, bandwidth, security group, and management. For detailed descriptions of each configuration item, see Configuration Item Descriptions.

6. Before you create the instance, review the overall configuration on the right side of the page and configure options such as the subscription duration. Ensure that all settings meet your requirements.

7. Click Confirm Order to create the instance.

   Instance creation typically takes 3 to 5 minutes. You can go to the Instances page in the console to check the status. When the instance status changes to Running, the instance is ready.

Step 2: Prepare the Kubernetes node

After creating the instance, connect to it and install the required software to configure it as a single-node Kubernetes cluster.

1. Install containerd To run containers, you must install containerd as the container runtime.

   # Update the package sources and install dependencies.
   sudo apt-get update
   sudo apt-get install -y ca-certificates curl
   
   # Add Docker's official GPG key.
   sudo install -m 0755 -d /etc/apt/keyrings
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   sudo chmod a+r /etc/apt/keyrings/docker.asc
   
   # Add Docker's APT repository.
   echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
     
   # Install containerd.
   sudo apt-get update
   sudo apt-get install -y containerd.io

2. Configure containerd Generate the default configuration file, and then modify key parameters for Kubernetes compatibility.

   # Generate the default configuration file.
   sudo mkdir -p /etc/containerd
   containerd config default | sudo tee /etc/containerd/config.toml
   
   # Replace the Kubernetes image registry with an Alibaba Cloud mirror to accelerate image pulls.
   sudo sed -i -E 's#registry.k8s.io#registry.aliyuncs.com/google_containers#g' /etc/containerd/config.toml
   
   # Change the cgroup driver to systemd to meet Kubernetes requirements.
   sudo sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
   
   # Restart the containerd service to apply the changes.
   sudo systemctl restart containerd

3. Configure node kernel parameters To meet the networking and runtime requirements for Kubernetes, disable the swap partition and load the necessary kernel modules.

   # Temporarily disable swap.
   sudo swapoff -a
   # Permanently disable swap by commenting out the swap entry in the fstab file.
   sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
   
   # Load the overlay and br_netfilter kernel modules.
   sudo modprobe overlay
   sudo modprobe br_netfilter
   
   # Configure the modules to load automatically at boot.
   sudo tee /etc/modules-load.d/k8s.conf <<EOF
   overlay
   br_netfilter
   EOF
   
   # Configure kernel parameters to allow IP forwarding and bridged traffic processing.
   sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
   net.bridge.bridge-nf-call-ip6tables = 1
   net.bridge.bridge-nf-call-iptables = 1
   net.ipv4.ip_forward = 1
   EOT
   
   # Apply all sysctl configurations.
   sudo sysctl --system

Step 3: Install a single-node Kubernetes cluster

Use the kubeadm tool to quickly set up a single-node Kubernetes cluster.

1. Install Kubernetes components Install kubeadm, kubelet, and kubectl.

   Ensure you use a Kubernetes version compatible with Confidential Containers. This document uses v1.32.0 as an example.

   # Add the GPG key for the Kubernetes APT repository.
   curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   
   # Add the Kubernetes APT repository.
   echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
   
   # Install the Kubernetes tools.
   sudo apt-get update
   sudo apt-get install -y kubectl kubeadm kubelet
   sudo apt-mark hold kubelet kubeadm kubectl

2. Initialize the Kubernetes control plane Use the kubeadm init command to initialize the control plane.

   # Get the IP address of the primary network interface to use as the API server's advertise address.
   # Note: This command assumes the primary interface is eth0. If your environment is different, replace it with the correct IP address.
   NODE_IP=$(ip -o -4 addr show dev eth0 | awk '{split($4,a,"/");print a[1]}')
   
   # Initialize the cluster.
   sudo kubeadm init --pod-network-cidr=10.10.0.0/16 \
     --apiserver-advertise-address ${NODE_IP} \
     --kubernetes-version v1.32.0 \
     --image-repository registry.aliyuncs.com/google_containers

   When initialization finishes, follow the output instructions to configure kubectl access credentials.

   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Install a network plugin Deploy the Flannel CNI plugin to enable pod-to-pod communication in the cluster.

   # Install the Flannel network plugin.
   export KUBECONFIG=/etc/kubernetes/admin.conf
   
   FLANNEL_VERSION=v0.27.4
   wget https://github.com/flannel-io/flannel/releases/download/${FLANNEL_VERSION}/kube-flannel.yml
   
   # Modify the Flannel manifest to set the correct pod network CIDR and use a regional image mirror.
   sed -i -E 's#([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]+#10.10.0.0/16#g' kube-flannel.yml
   sed -i -E 's#ghcr.io/flannel-io#confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product#g' kube-flannel.yml
   
   # Deploy Flannel.
   kubectl apply -f kube-flannel.yml

4. Remove the control plane taint To allow workloads to be scheduled on this single-node cluster, remove the NoSchedule taint from the control plane node.

   NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
   kubectl taint nodes $NODE_NAME node-role.kubernetes.io/control-plane-

Step 4: Deploy CoCo components

Confidential Containers offers multiple deployment methods. Choose either the Helm chart method or the Operator method.

mkdir -p kustomize && cd kustomize

mkdir release && mkdir -p ccruntime/default

cat <<EOF > release/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "github.com/confidential-containers/operator/config/release?ref=v0.17.0"

images:
- name: quay.io/confidential-containers/operator
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-operator
EOF

cat <<EOF > ccruntime/default/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.17.0"

images:
- name: quay.io/confidential-containers/reqs-payload
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-reqs-payload
- name: quay.io/kata-containers/kata-deploy-ci
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-kata-deploy-ci
- name: quay.io/kata-containers/kata-deploy
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-kata-deploy
EOF

# Allow the current node to act as a worker.
for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

kubectl apply -k release
kubectl apply -k ccruntime/default

Step 5: Deploy and verify a sample pod

After deploying all components, run a sample pod and verify that it is running in a TDX confidential VM.

1. Label the node Add a label to the worker node to indicate that it supports the Kata runtime.

   NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
   kubectl label node $NODE_NAME katacontainers.io/kata-runtime=true

2. Deploy a sample pod Create a pod and specify the TDX runtime in its spec by using runtimeClassName.

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: coco-demo-pod
   spec:
     runtimeClassName: kata-qemu-tdx
     containers:
       - image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
         name: hello-alinux
         command:
           - "sleep"
           - "infinity"
   EOF

3. Verify the deployment status Check that the pod is running successfully and confirm that it is using the correct runtime class.

   # Wait for the pod status to become Running.
   kubectl get pod coco-demo-pod
   
   # Describe the pod and confirm that the "Runtime Class" field is "kata-qemu-tdx".
   kubectl describe pod coco-demo-pod | grep "Runtime Class"

4. Verify the TDX environment Log in to the bare metal instance and check the kernel logs to confirm that the TDX module is initialized and in use.

   # Run this command on the node.
   dmesg | grep -i tdx

   If you see output containing TDX module initialized or similar TDX-related messages, the TDX environment is activated, and the Kata runtime is using this hardware feature to create confidential VMs.