Community Blog Dragonfly Promoted to a CNCF Incubation Project

Dragonfly Promoted to a CNCF Incubation Project

This article provides an overview of the Dragonfly project and discusses how it is promoted from a sandbox level project to a CNCF incubation project.


Dragonfly is a cloud-native open-source image and file distribution system developed by Alibaba Cloud to solve cloud-native image distribution in Kubernetes-centered applications. Recently, the Technical Oversight Committee (TOC) of the Cloud Native Computing Foundation (CNCF) voted to promote Dragonfly as the incubation project.

In October 2018, Dragonfly officially became a CNCF sandbox level project and started its cloud-native journey to help users improve the experience of distributing images and files in Kubernetes. It allows enterprise engineers to focus on applications rather than infrastructure management. To learn more about Dragonfly, click here.

According to Yi Li, a senior technical expert at Alibaba Cloud, "as one of the key technologies across the container platform in the Alibaba ecosystem, Dragonfly supports the deployment and delivery of billions of application services every year and is used by many enterprises and customers around the world. Alibaba looks forward to continuously improving Dragonfly to make it more efficient and easier to use."

Dragonfly is designed to solve distribution problems in cloud-native scenarios. The project consists of the following three key parts:

  • The SuperNode acts as a central scheduler and controls all the distribution processes among peers.
  • The DFget is a peer-to-peer (P2P) client, which is mainly responsible for block transfer between peers.
  • The Dfdaemon acts as a proxy, intercepts image download requests from container engines, and redirects the requests to DFget.

In the words of TOC member and project sponsor, Dr. Liang Sheng, "Dragonfly improves the efficiency of image and file distribution by using P2P technologies and improves user experience by reducing the network load of the image warehouse. As organizations around the world migrate workloads to container stacks, we expect Dragonfly adoption to keep on increasing significantly."

Dragonfly is currently integrated with other CNCF projects, including Prometheus, Containerd, Harbor, Kubernetes, and Helm. The maintainers of the project come from Alibaba, ByteDance, eBay, and Meitu. The project contributors come from more than 20 companies, including NetEase, Jingdong, Wal-Mart, VMware, Shopee, China Mobile, Qunar, Qiniu, and NVIDIA.

Dragonfly: Main Features

  • P2P-based file distribution: By transferring files with P2P technologies, Dragonfly may fully use the bandwidth of each peer to improve the download efficiency and significantly save the IDC bandwidth, especially the expensive cross-region and cross-nation bandwidth.
  • Unintrusive support for various container technologies: Dragonfly seamlessly supports the distribution of images by various containers, such as Docker and containerd.
  • Host-level rate limit: Many download tools (wget or curl) only set a rate limit on the current download task, while Dragonfly sets a rate limit on the entire host.
  • Passive CDN: The Passive CDN mechanism avoids repeated remote downloads.

Important Milestones

  • Maintainers from 7 projects in 4 organizations
  • 67 contributors
  • 21 contributing organizations
  • 4.6k+ GitHub Stars
  • More than 100,000 downloads on Docker Hub
  • 120% increase in commits last year

Dragonfly witnessed a significant journey since entering the CNCF sandbox stage and has become more cloud-native.

  • It is refactored by using Golang to be friendlier to developers and easier to integrate with CNCF.
  • It actively integrates with Kubernetes, Harbor, containerd, Prometheus, and Helm.
  • With the help of CNCF, Dragonfly strived to help the entire industry adopt cloud-native distribution.

According to the CNCF CTO and COO Chris Aniszczyk, "Distributing container images in a large-scale production environment has become a major challenge to be addressed as cloud-native applications continue to grow. We are very pleased that Dragonfly has shared its practical experience at Alibaba. The Dragonfly project has recently been completely rewritten in Golang and has made great progress. We look forward to cultivating more projects like Dragonfly and making our projects more diversified."

Dragonfly's latest version, 1.0.0 has been fully rewritten with Golang to improve its usability with other cloud-native technologies. Now, Dragonfly brings a more flexible and scalable architecture, on which more cloud solutions and potential integration with Open Container Initiative (OCI) make image distribution more efficient. Sun Hongliang, a senior technical expert at Alibaba and the Dragonfly project maintainer, said, "We are delighted that Dragonfly has become an incubation project. As maintainers, we are also working hard to improve this project in all aspects. We look forward to unleashing more possibilities with Dragonfly."

Dragonfly Implementation

Since entering the CNCF sandbox stage, Dragonfly has grown rapidly in various industries, including e-commerce, telecommunications, finance, and the Internet. It is also used by various customers, including Alibaba, China Mobile, Shopee, Bilibili, Ant Financial, Huya, Didi.

China Mobile Zhejiang branch has adopted Dragonfly in the production environment for more than three years, involving more than 1,000 physical computers. Currently, they are running more than 200 business systems and 1,700 application modules on Dragonfly.

Shopee is a Singapore E-commerce platform under the Sea Group with its business across Malaysia, Thailand, Taiwan, Indonesia, Vietnam, and the Philippines, and offers customers a simple, secure, and fast way to shop online. Shopee has used Dragonfly for more than one year in the production environment, involving more than 10,000 physical machines.

The Chinese video comment website, Bilibili, has adopted Dragonfly in the test and production environments of more than 3,900 machines. Furthermore, engineers from Bilibili work with and make contributions to the Dragonfly community on registry verification and stability.

Alibaba Cloud Container Registry Enterprise Edition (ACR EE) is a leading cloud-native asset management platform in China. It provides secure management and efficient distribution of products such as Docker or OCI images and Helm charts. It is widely used by top enterprises in many industries. In combination with Dragonfly's large-scale image distribution capability, ACR EE ensures rapid scaling and deployment of customers' container services. The ACR team works with the community to promote Dragonfly's continuous optimization based on enterprise cloud-native scenarios and cloud environments.

Promising Future

In the future, Dragonfly will consolidate the file and image distribution capability and move towards a more flexible and scalable architecture. In addition, the Dragonfly community will work closely with Linux Foundation OCI to make image distribution more efficient:

  • Actively explore technological innovations based on industry scenarios:

1) Use machine learning algorithms to support more intelligent P2P scheduling and further improve distribution efficiency.
2) Strengthen data security and integrity during data transmission and improve the transmission efficiency.
3) Encapsulate Dragonfly functions into plug-ins and categorize them as scenarios, based on cloud-native infrastructure, in order to flexibly adapt to various fields such as Bare Metal, Cloud Servers, and IoT.

  • Collaborate with the OCI community to formulate image and file distribution standards for the cloud-native field, and provide integration capabilities by opening the data distribution interfaces.
0 0 0
Share on

You may also like


Related Products