Elastic Compute Service:Deploy CoCo on Alibaba Cloud bare metal

Step 1: Create a TDX bare metal instance

Create an 8th-generation Alibaba Cloud bare metal instance with Intel TDX support.

1. Go to the instance purchase page.

2. Select a billing method, region, and availability zone. The required community image is available only in specific zones. For this example, select Beijing Zone I.

3. In the Instance section, click Elastic Bare Metal Server and select the ecs.ebmg8i.48xlarge instance type.

4. In the Image section, select Community Image and search for image ID m-2ze2ucup4c5bvgx751lx. Select the Confidential VM checkbox to enable TDX.

   Important

   This is a custom Ubuntu-based image with pre-installed TDX drivers. The default SSH user is ubuntu, not root.

5. Configure the network, storage, bandwidth, security group, and management settings. See Configuration Item Descriptions.

6. Review the overall configuration on the right side of the page. Configure options such as the subscription duration and verify that all settings meet your requirements.

7. Click Confirm Order to create the instance.

   Instance creation takes 3 to 5 minutes. Go to the Instances page in the console to check the status. The instance is ready when its status changes to Running.

Step 2: Prepare the Kubernetes node

Connect to the instance and install the required software to set it up as a single-node Kubernetes cluster.

1. Install containerd

   # Update the package sources and install dependencies.
   sudo apt-get update
   sudo apt-get install -y ca-certificates curl
   
   # Add Docker's official GPG key.
   sudo install -m 0755 -d /etc/apt/keyrings
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   sudo chmod a+r /etc/apt/keyrings/docker.asc
   
   # Add Docker's APT repository.
   echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
     
   # Install containerd.
   sudo apt-get update
   sudo apt-get install -y containerd.io

2. Configure containerd Generate the default configuration file and modify key parameters for Kubernetes compatibility.

   # Generate the default configuration file.
   sudo mkdir -p /etc/containerd
   containerd config default | sudo tee /etc/containerd/config.toml
   
   # Replace the Kubernetes image registry with an Alibaba Cloud mirror to accelerate image pulls.
   sudo sed -i -E 's#registry.k8s.io#registry.aliyuncs.com/google_containers#g' /etc/containerd/config.toml
   
   # Change the cgroup driver to systemd to meet Kubernetes requirements.
   sudo sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
   
   # Restart the containerd service to apply the changes.
   sudo systemctl restart containerd

3. Configure kernel parameters Disable the swap partition and load the required kernel modules for Kubernetes networking and runtime.

   # Temporarily disable swap.
   sudo swapoff -a
   # Permanently disable swap by commenting out the swap entry in the fstab file.
   sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
   
   # Load the overlay and br_netfilter kernel modules.
   sudo modprobe overlay
   sudo modprobe br_netfilter
   
   # Configure the modules to load automatically at boot.
   sudo tee /etc/modules-load.d/k8s.conf <<EOF
   overlay
   br_netfilter
   EOF
   
   # Configure kernel parameters to allow IP forwarding and bridged traffic processing.
   sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
   net.bridge.bridge-nf-call-ip6tables = 1
   net.bridge.bridge-nf-call-iptables = 1
   net.ipv4.ip_forward = 1
   EOT
   
   # Apply all sysctl configurations.
   sudo sysctl --system

Step 3: Install a single-node Kubernetes cluster

Use kubeadm to set up a single-node Kubernetes cluster.

1. Install Kubernetes components Install kubeadm, kubelet, and kubectl.

   # Add the GPG key for the Kubernetes APT repository.
   curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   
   # Add the Kubernetes APT repository.
   echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
   
   # Install the Kubernetes tools.
   sudo apt-get update
   sudo apt-get install -y kubectl kubeadm kubelet
   sudo apt-mark hold kubelet kubeadm kubectl

2. Initialize the control plane Use kubeadm init to initialize the control plane.

   # Get the IP address of the primary network interface to use as the API server's advertise address.
   # Note: This command assumes the primary interface is eth0. If your environment is different, replace it with the correct IP address.
   NODE_IP=$(ip -o -4 addr show dev eth0 | awk '{split($4,a,"/");print a[1]}')
   
   # Initialize the cluster.
   sudo kubeadm init --pod-network-cidr=10.10.0.0/16 \
     --apiserver-advertise-address ${NODE_IP} \
     --kubernetes-version v1.32.0 \
     --image-repository registry.aliyuncs.com/google_containers

   After initialization, follow the output instructions to configure kubectl access credentials.

   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Install a network plugin Deploy the Flannel CNI plugin for pod-to-pod communication.

   # Install the Flannel network plugin.
   export KUBECONFIG=/etc/kubernetes/admin.conf
   
   FLANNEL_VERSION=v0.27.4
   wget https://github.com/flannel-io/flannel/releases/download/${FLANNEL_VERSION}/kube-flannel.yml
   
   # Modify the Flannel manifest to set the correct pod network CIDR and use a regional image mirror.
   sed -i -E 's#([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]+#10.10.0.0/16#g' kube-flannel.yml
   sed -i -E 's#ghcr.io/flannel-io#confidential-ai-registry.cn-shanghai.cr.aliyuncs.com/product#g' kube-flannel.yml
   
   # Deploy Flannel.
   kubectl apply -f kube-flannel.yml

4. Remove the control plane taint Allow workloads on this single-node cluster by removing the NoSchedule taint from the control plane node.

   NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
   kubectl taint nodes $NODE_NAME node-role.kubernetes.io/control-plane-

Step 4: Deploy CoCo components

Choose either the Helm chart or Operator method to deploy CoCo.

mkdir -p kustomize && cd kustomize

mkdir release && mkdir -p ccruntime/default

cat <<EOF > release/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "github.com/confidential-containers/operator/config/release?ref=v0.17.0"

images:
- name: quay.io/confidential-containers/operator
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-operator
EOF

cat <<EOF > ccruntime/default/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.17.0"

images:
- name: quay.io/confidential-containers/reqs-payload
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-reqs-payload
- name: quay.io/kata-containers/kata-deploy-ci
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-kata-deploy-ci
- name: quay.io/kata-containers/kata-deploy
  newName: registry-cn-hangzhou.ack.aliyuncs.com/dev/coco-kata-deploy
EOF

# Allow the current node to act as a worker.
for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

kubectl apply -k release
kubectl apply -k ccruntime/default

Step 5: Deploy and verify a sample pod

Run a sample pod and verify that it runs in a TDX confidential VM.

1. Label the node Label the worker node to indicate Kata runtime support.

   NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
   kubectl label node $NODE_NAME katacontainers.io/kata-runtime=true

2. Deploy a sample pod Create a pod that uses the TDX runtime via runtimeClassName in its spec.

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: coco-demo-pod
   spec:
     runtimeClassName: kata-qemu-tdx
     containers:
       - image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
         name: hello-alinux
         command:
           - "sleep"
           - "infinity"
   EOF

3. Verify the deployment Confirm that the pod is running with the correct runtime class.

   # Wait for the pod status to become Running.
   kubectl get pod coco-demo-pod
   
   # Describe the pod and confirm that the "Runtime Class" field is "kata-qemu-tdx".
   kubectl describe pod coco-demo-pod | grep "Runtime Class"

4. Verify TDX activation Check the kernel logs on the bare metal instance to confirm TDX initialization.

   # Run this command on the node.
   dmesg | grep -i tdx

   Output containing TDX module initialized or similar messages confirms that the TDX environment is active and the Kata runtime is using it for confidential VMs.