Kubernetes experience of containerd and sandbox-Alibaba Cloud Developer Community

containerd is an open-source industry standard container runtime that focuses on simplicity, stability, and portability. It also supports Linux and Windows. On December 14, 2016, Docker announced that it would donate the core component containerd of Docker Engine to a new open-source community for independent development and operation. Alibaba Cloud, AWS, Google,IBM, and Microsoft are the initial members to jointly build the containerd community. In March 2017, Docker donated containerd to CNCF (Cloud Native Computing Foundation). containerd is rapidly developed and widely supported. The Docker engine has already used containerd as the basis for container lifecycle management. Kubernetes, in May 2018, containerd was officially supported as the Container Runtime Manager. In February 2019, CNCF announced that it graduated from containerd and became a project available for production.

containerd has built-in Container Runtime Interface (CRI) support since version 1.1, further simplifying the support for Kubernetes. The architecture diagram is as follows:

in Kubernetes scenarios, containerd has a lower resource usage and a faster startup speed than full Docker Engine.

SOURCE containerd

cri-o dominated by Red Hat is a container runtime management project that competes with containerd. Compared with cri-o projects, containerd has performance advantages and is more widely supported in the community.

Share from ebay

more importantly, containerd provides a flexible extension mechanism to support the runtime implementation of various OCI(Open Container Initiative) containers, such as runc containers (also known as Docker containers),KataContainer, security Sandbox containers such as gVisor and Firecraker.

In Kubernetes environment, you can use different APIs and command line tools to manage containers, pods, and images. The following figure shows how to use different levels of APIs and CLI to manage container lifecycle management.

  • Kubectl: it is a command line tool at the cluster level and supports the basic concepts of Kubernetes.
  • Lifectl: a command line tool for CRI on a node.
  • ctr: a command line tool for containerd.


Minikube is the easiest way to experience containerd as a Kubernetes Container Runtime. We will use containerd as a Kubernetes Container Runtime and support runc and gvisor.

In the early days, due to network access, many friends were unable to directly use official Minikube for experiments. The latest version of Minikube 1.5 provides a complete configuration method to help you obtain the required Docker images and configurations by using the Alibaba Cloud Image address, container Runtime is supported, such as Docker and Containerd.

We create a Minikube virtual machine environment. For more information, see https://yq.aliyun.com/articles/221687. Note that --container-runtime=containerdthe containerd parameter is used as the container runtime. You must also replace registry-mirror with your Alibaba cloud image acceleration address.

$ minikube start --image-mirror-country cn \
    --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.5.0.iso \
    --registry-mirror=https://XXX.mirror.aliyuncs.com \

  Darwin 10.14.6 上的 minikube v1.5.0
  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
️  您所在位置的已知存储库都无法访问。正在将 registry.cn-hangzhou.aliyuncs.com/google_containers 用作后备存储库。
  正在创建 hyperkit 虚拟机(CPUs=2,Memory=2000MB, Disk=20000MB)...
️  VM is unable to connect to the selected image repository: command failed: curl -sS https://k8s.gcr.io/
stderr: curl: (7) Failed to connect to k8s.gcr.io port 443: Connection timed out
: Process exited with status 7
  正在 containerd 1.2.8 中准备 Kubernetes v1.16.2…
  拉取镜像 ...
  正在启动 Kubernetes ...
⌛  Waiting for: apiserver etcd scheduler controller
  完成!kubectl 已经配置至 "minikube"

$ minikube dashboard
  Verifying dashboard health ...
  Launching proxy ...
  Verifying proxy health ...
  Opening in your default browser...

Deploy test applications

deploy an nginx application through a Pod

$ cat nginx.yaml
apiVersion: v1
kind: Pod
  name: nginx
  - name: nginx
    image: nginx
$ kubectl apply -f nginx.yaml
pod/nginx created

$ kubectl exec nginx -- uname -a
Linux nginx 4.19.76 #1 SMP Fri Oct 25 16:07:41 PDT 2019 x86_64 GNU/Linux

then, enable minikube to support gvisor

$ minikube addons enable gvisor
  gvisor was successfully enabled

$ kubectl get pod,runtimeclass gvisor -n kube-system
pod/gvisor   1/1     Running   0          60m

NAME                              CREATED AT
runtimeclass.node.k8s.io/gvisor   2019-10-27T01:40:45Z

$ kubectl get runtimeClass
gvisor   2019-10-27T01:40:45Z

when gvisorpod entry Runningin the status, you can deploy the gvisor test application.

You can see that a gvisor"runtimeClassName".

After that, developers can select different types of Container Runtime implementations by using the runtimeClassName in the Pod declaration. For example, create an nginx application running in the gvisor sandbox container.

$ cat nginx-untrusted.yaml
apiVersion: v1
kind: Pod
  name: nginx-untrusted
  runtimeClassName: gvisor
  - name: nginx
    image: nginx

$ kubectl apply -f nginx-untrusted.yaml
pod/nginx-untrusted created

$ kubectl exec nginx-untrusted -- uname -a
Linux nginx-untrusted 4.4 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

We can clearly find that because runc-based containers share the operating system kernel with the host, the OS kernel version displayed in the runc container is the same as that of the Minikube host. The runsc container of the gvisor uses an independent kernel, which is different from the OS kernel version of the Minikube host.

Each sandbox container has an independent kernel, which reduces the security attack surface and provides better security isolation features. It is suitable for isolation of untrusted applications or multi-tenant scenarios.

Note: in minikube, the gvisor intercepts kernel calls through ptrace, resulting in high performance loss. In addition, gvisor compatibility needs to be enhanced.

Use ctl and hadotl tools

now we can enter Minikube virtual machine

$ minikube ssh

containerd allows you to isolate container resources by namespace to view existing containerd namespaces.

$ sudo ctr namespaces ls
# 列出所有容器镜像
$ sudo ctr --namespace=k8s.io images ls

# 列出所有容器列表
$ sudo ctr --namespace=k8s.io containers ls

A simpler way to Kubernetes environment is to use crictlperform operations on pods

# 查看pod列表
$ sudo crictl pods
POD ID              CREATED             STATE               NAME                                         NAMESPACE              ATTEMPT
78bd560a70327       3 hours ago         Ready               nginx-untrusted                              default                0
94817393744fd       3 hours ago         Ready               nginx                                        default                0

# 查看名称包含nginx的pod的详细信息
$ sudo crictl pods --name nginx -v
ID: 78bd560a70327f14077c441aa40da7e7ad52835100795a0fa9e5668f41760288
Name: nginx-untrusted
UID: dda218b1-d72e-4028-909d-55674fd99ea0
Namespace: default
Status: Ready
Created: 2019-10-27 02:40:02.660884453 +0000 UTC
    io.kubernetes.pod.name -> nginx-untrusted
    io.kubernetes.pod.namespace -> default
    io.kubernetes.pod.uid -> dda218b1-d72e-4028-909d-55674fd99ea0
    kubectl.kubernetes.io/last-applied-configuration -> {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"nginx-untrusted","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"nginx"}],"runtimeClassName":"gvisor"}}

    kubernetes.io/config.seen -> 2019-10-27T02:40:00.675588392Z
    kubernetes.io/config.source -> api

ID: 94817393744fd18b72212a00132a61c6cc08e031afe7b5295edafd3518032f9f
Name: nginx
UID: bfcf51de-c921-4a9a-a60a-09faab1906c4
Namespace: default
Status: Ready
Created: 2019-10-27 02:38:19.724289298 +0000 UTC
    io.kubernetes.pod.name -> nginx
    io.kubernetes.pod.namespace -> default
    io.kubernetes.pod.uid -> bfcf51de-c921-4a9a-a60a-09faab1906c4
    kubectl.kubernetes.io/last-applied-configuration -> {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"nginx","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"nginx"}]}}

    kubernetes.io/config.seen -> 2019-10-27T02:38:18.206096389Z
    kubernetes.io/config.source -> api

for more information, see


relationship between containerd and Docker

many people are concerned about the relationship between containerd and Docker, and whether containerd can replace Docker? containerd has become the mainstream implementation of Container Runtime and is strongly supported by the Docker community and the Kubernetes community. Docker Engine container lifecycle management at the underlying layer is also implemented based on containerd.

However, Docker Engine contains more developer toolchains, such as image building. It also includes Docker's own log, storage, network, and Swarm orchestration capabilities. In addition, most container ecosystem vendors, such as security, monitoring, and development, support for Docker Engine and support for containerd.

Therefore, in Kubernetes runtime environment, users who pay more attention to security, efficiency and customization can choose containerd as the container runtime environment. For most developers, continuing to use Docker Engine as a container runtime is also a good choice.

Alibaba Cloud container service supports containerd

In Alibaba Cloud Kubernetes service ACK, we have adopted containerd as the container runtime management to support the hybrid deployment of sandboxed containers and runc containers. Among the existing products, we, together with the Alibaba cloud operating system team and Ant Financial, support the runV sandbox container based on lightweight virtualization. 4Q will also work with the operating system Team, the security team released a Intel SGX-based trusted encrypted sandbox container.

For more information, see https://help.aliyun.com/document_detail/140541.html.

And Serverless Kubernetes(ASK), we also use the containerd flexible plug-in mechanism to customize and tailor the container runtime implementation for the nodeless environment.

Selected, One-Stop Store for Enterprise Applications
Support various scenarios to meet companies' needs at different stages of development

Start Building Today with a Free Trial to 50+ Products

Learn and experience the power of Alibaba Cloud.

Sign Up Now