In the production practice of containers, smaller container images can be deployed and started quickly. When the application image reaches several gigabytes or above, it usually takes a lot of time to download the images on the node. Dragonfly, a CNCF incubating project, improves the efficiency of large-scale distribution of container images by introducing P2P networks. However, users still have to wait for the complete image data to be downloaded locally before creating their containers. Therefore, we hope to reduce the time taken to download images so users can deploy container applications faster. At the same time, how to better protect users' data is also an important concern of the container industry in recent years.
To this end, we introduced a container image acceleration service called Nydus for the Dragonfly project. Nydus can shorten the image download time and provide end-to-end image data consistency verification so users can manage container applications more safely and quickly. Nydus is jointly developed by Alibaba Cloud and Ant Group engineers and deployed in the internal production environment on a large scale. As part of the cloud-native ecosystem, Nydus has shown excellent performance in the production environment. This has given us the confidence to make the project open-source. This way, more container users can experience the capabilities of quick startup and secure loading of containers.
The Nydus project implements a user-mode file system on top of a container image format that improves over the current OCI image specification. These optimizations give Nydus the following features:
In terms of architecture, Nydus mainly includes a new image format and the Filesystem in Userspace (FUSE) responsible for parsing container images.
Nydus can parse the FUSE or virtiofs protocols to support a traditional runc container or Kata container. Container repositories, OSS, NAS, and the super nodes and peer nodes of Dragonfly can all be used as image data sources for Nydus. At the same time, Nydus can also configure a local cache to avoid pulling data from remote data sources every time it is started.
In terms of image format, Nydus divides a container image into two layers: metadata and data. The metadata layer is a self-verification hash tree. Each file and directory is a node with a hash value in the hash tree. The hash value of a file node is determined by the data of the file, and the hash value of a directory node is determined by the hash value of all files and directories under the directory. The data of each file is sliced according to a fixed size and saved to the data layer. Data slices can be shared in different files and in different files of different images.
If a user deploys the Nydus image service, one of the most visible improvements is less time is spent on container startup than before. It changes from a long startup in the past to an instantaneous one. In our testing, Nydus shortened the startup time of common images from minutes to seconds.
Another less obvious (but also important) improvement is that Nydus can provide users with container runtime data consistency verification. In traditional images, image data is first decompressed to the local file system and accessed by container applications. Before decompression, the image data integrity is checked. However, the image data integrity can no longer be checked after decompression. There is one problem here. If the decompressed image data is unintentionally or maliciously modified, the user cannot perceive it. With Nydus, the image will not be decompressed to the local, and integrity checks can be performed for each data access. If the data is tampered with, it can be pulled again from the remote data source.
We have introduced the architecture and advantages of Nydus. Over the past year, we have worked with our internal product team to make Nydus more stable, safe, and easy to use. After making Nydus open-source, we will make more efforts to adapt to the cloud-native container ecosystem. Our vision is that when users deploy Dragonfly and Nydus services in a cluster, they can easily and quickly run their container applications regardless of the size of the image and do not need to worry about the data security of the container image.
We have deployed Nydus on a large scale in our internal production environment. We believe improvements to OCI image specifications require extensive community efforts. Therefore, we actively participated in the OCI community's discussion on the next-generation image specification and found that Nydus can meet the OCI community's requirements for the next-generation image format in a wide range of aspects. Therefore, we propose implementing Nydus as an example of the next-generation image specification in the OCI community. We look forward to working with more cloud-native industry leaders to promote the formulation and implementation of the next-generation image specification.
Q: What are the problems with the existing OCI image specification?
Q: What is the difference between Nydus and CRFS?
Q: What is the difference between Nydus and Azure Teleport?
Q: What if the network is broken when running Nydus-based containers?
In June 2020, the OCI community spent roughly a month discussing the shortcomings of the current OCI image specification and which requirements the OCIv2 image format needs to meet. OCIv2 is an improvement of the current OCI image specification, not a new image specification.
This great discussion on image format started with an email and a shared document. It also led to many online OCI community discussion meetings. The conclusion is also very encouraging. The OCIv2 image format needs to meet the following requirements:
You can find a detailed description of each requirement in this shared document. We participated in the whole discussion of the OCIv2 image format requirements and found that Nydus met all these requirements well. This prompted us to make the Nydus project open-source to provide a basis for community discussions.
This articlce was originally published on CNCF wechat official account, Dragonfiy.
OpenAnolis - January 10, 2023
OpenAnolis - January 11, 2024
Alibaba Developer - July 20, 2021
OpenAnolis - June 15, 2022
Alibaba Cloud Community - January 13, 2023
OpenAnolis - April 7, 2023
Provides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resourcesLearn More
Multi-source metrics are aggregated to monitor the status of your business and services in real time.Learn More
Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.Learn More
A secure image hosting platform providing containerized image lifecycle managementLearn More
More Posts by OpenAnolis