Dragonfly P2P-based file and image distribution system

Background

Network download

When it comes to network download, you should first think of the C/S mode based on TCP/IP protocol cluster. In this mode, each client is expected to establish a TCP connection with the server. The server polls and listens for TCP connections and responds in turn, as shown in the following figure:

At the end of last century, based on the idea of C/S mode, people developed HTTP, FTP and other application layer protocols. However, the disadvantages of C/S mode are obvious: the server load is too large and the download rate is too slow. With the increase of the scale of the Internet and the increase of customers' demand for download data size and download speed, these drawbacks are constantly amplified.

P2P download principle

Based on the above background, some people put forward P2P download mode combining the idea of P2P network and load balancing. This mode no longer puts all the download pressure on the server. The server is only responsible for transferring file metadata. The real file download connection is established between the client and the client. At the same time, a file can be divided into multiple blocks, and different blocks in the same file can be downloaded on different clients, which makes the downloaded files circulate dynamically in the P2P network and greatly improves the download efficiency, as shown in the following figure:

Decentralized P2P download is based on DHT technology, which uses distributed whole network mode to store and retrieve information. All information is stored in the form of hash table entries, which are distributed on each node, thus forming a huge distributed hash table in a network-wide manner. On this basis, the decentralization of a single server is achieved. The hash table is responsible for the allocation of load, and the load of the whole network is evenly distributed to multiple machines.

Dragonfly Introduction and Architecture Overview

Dragonfly is a P2P based intelligent image and file distribution tool. It aims to improve the efficiency and speed of large-scale file transfer and maximize the use of network bandwidth. It is widely used in application distribution, cache distribution, log distribution and image distribution.

Principle

Dragonfly combines the advantages of C/S architecture and P2P architecture. It provides a customer-oriented C/S architecture download mode. At the same time, it also provides a peer-to-peer back-to-source mode for server clusters. Unlike traditional P2P, peer-to-peer networks are built inside the Scheduler, with the goal of maximizing the internal download efficiency of P2P, as shown in the following figure:

Introduction to architecture

Dragonfly is oriented to image distribution and file distribution, and combines the idea of P2P network and server cluster to provide users with stable and efficient download services. Dragonfly hopes to build a P2P network inside the server, and divide the different host nodes of the server into four roles: Manager, Scheduler, Seed Peer and Peer, providing different functions respectively.

The Manager provides the overall configuration function, pulls the configuration of other roles and communicates with each other. The Scheduler provides the download scheduling function, and its scheduling results directly affect the download rate. Seed Peer is responsible for downloading back to the source and pulling the required image or file from the external network. As a server in the C/S architecture, Peer provides download functions to customers through various protocols. The architecture diagram is as follows:

Among them, Seed Peer supports the use of multiple protocols to download from the external network back to the source, and also supports the use as a peer in the cluster. Peer provides download services based on multiple protocols, as well as proxy services for image warehouses or other download tasks.

Component Details

Manager

Manager plays the role of manager when deploying multiple P2P clusters, providing front-end console for users to operate P2P clusters visually. It mainly provides dynamic configuration management, maintenance of cluster stability, and maintenance of association relationships among multiple P2P clusters. For maintaining the overall stability of the cluster, the Manager and each service maintain Keepalive to ensure that abnormal instances can be eliminated in case of instance exceptions. Dynamic configuration management can operate the control units of various components on the Manager, such as controlling the load of the Peer and Seed Peer, and the number of parents scheduled by the Scheduler. The Manager can also maintain multiple sets of P2P cluster associations. A Scheduler Cluster, a Seed Peer Cluster and several Peers form a complete P2P cluster. Of course, different P2P clusters can be network isolated. Under normal circumstances, a set of P2P clusters in one computer room is used, and multiple P2P clusters are managed by one manager.

Scheduler

The main task of the Scheduler is to find the optimal parent node for the current download node and trigger the Seed Peer to download back to the source. Let Peer download back to the source when appropriate. When the scheduler starts, it first registers with the Manager. After the registration is successful, it initializes the dynamic configuration client, pulls the dynamic configuration from the Manager, and then starts the services required by the scheduler itself.

The core of the Scheduler is to select a set of optimal Parent nodes for the current download peer to download. The Scheduler faces tasks. A task is a complete download task. The Scheduler stores the task information and the DAG of the corresponding P2P download network. The scheduling process is to first filter the abnormal Parent node and filter it according to multi-dimensions. For example, to judge whether the Peer is a BadNode, the judgment logic is to assume that the response time of each node follows the normal distribution. If the current response time of a node is 6 σ If it is outside the range, the node is considered as BadNode, and the node is deleted. Then score the remaining pending parent nodes according to the historical download characteristic value, and return a group of parents with the highest score to the current peer for download.

Seed Peer and Peer

Seed Peer and Peer have many similarities. They are all based on Dfdaemon. The difference is that Seed Peer adopts the Seed Peer mode and supports active triggering of back-to-source download. Peer adopts the Peer mode. As a server in the C/S architecture, it provides users with download function, and supports the passive trigger of the Scheduler to download back to the source. This indicates that the relationship between the peer and the seed peer is not fixed. A peer can make itself a seed peer by returning to the source. The seed peer can also change its running state to peer. The scheduler will dynamically change the corresponding DAG. In addition, both Seed Peer and Peer need to participate in the scheduling download process. The Scheduler may select Seed Peer or Peer as the parent node to provide the download function to other peers.

Dfstore and Dfcache

Dfcache is the cache client of Dragonfly. It communicates with dfdaemon and operates on files in P2P network, where P2P network acts as the cache system. You can store the corresponding Tasks and DAGs in the Scheduler.

Dfstore is a dragonfly storage client It can rely on different types of object storage services as backends to provide stable storage solutions, and now supports S3 and OSS. Dfstore relies on Backend object storage service combined with the acceleration characteristics of P2P itself. It can achieve fast writing and fast reading, save back-to-source and cross-machine room traffic, and reduce the pressure of the source station.

Advantage

Stability

Dragonfly will automatically isolate abnormal nodes to improve download stability. Each component in Dragonfly will contact the Manager through Keepalive. The Manager can ensure that the Scheduler address returned to the Peer and the Seed Peer address returned to the Scheduler are available. The unavailable Scheduler and Seed Peer will not be pushed by the Manager to the Peer or Scheduler that needs to download the task, so as to achieve the purpose of isolating exception nodes. This is also the exception isolation of the instance dimension, as shown in the following figure:

In addition, Dragonfly takes Task as the unit during scheduling, which also ensures the stability of the whole scheduling process. After receiving a new Task scheduling request, the Scheduler triggers the Seed Peer to download back to the source; After receiving a scheduling request for an existing task, the Scheduler schedules the optimal Parent Peer set and returns it to the Peer. This logic ensures that Dragonfly can process the task whether it has been downloaded or not. In addition, in the scheduling process of the Scheduler, the peer whose response time is too slow will not be returned as a parent peer because it is considered as an exception node at present. This is also the exception isolation of the Task dimension.

Efficiency

Dragonfly uses P2P to carry out the internal back-to-source of the server. The P2P download itself allocates the load and minimizes the load of each server node. The following details ensure the efficiency of Dragonfly download:

• The Scheduler returns the current locally optimal parent set to Peer by scoring each possible parent, and Peer downloads based on this set.

• The download process is based on Task. Each Task divides the file to be downloaded into multiple pieces of Pieces. After Peer gets the best Parent, it broadcasts the download request of each Piece to this collection. The Parent in the collection returns the meta information of the corresponding Piece to Peer after receiving the request. Peer uses the Parent Peer corresponding to the first received Piece meta information as the actual download source of the Piece. This approach takes into account the possible changes in the period from the time the Scheduler returns the available Parent to the time it triggers the download, and allows Peer to obtain data from different download sources for different Pieces.

• Dfdaemon is divided into Seed Peer mode and Peer mode. It allows Seed Peer and Peer to switch. The number of machines serving as Seed Peer and Peer can be changed according to the actual needs, and dynamic adjustment is more suitable for the actual situation.

Easy to use

Dragonfly provides multiple deployment methods of Helm Charts, Docker Compose, Docker Image and binary. Users can quickly deploy a simple POC with one click, and can also deploy large-scale production based on Helm Charts. Of course, Dragonfly's various services have complete Metrics and ready-made Granafa templates to facilitate users to observe P2P traffic trends.

Other

Dragonfly, as the standard solution of CNCF in the field of image acceleration, can maximize the speed of image download by combining the Dragonfly sub-project Nydus with on-demand loading. In the future, we will continue to work hard to build an ecological chain in the field of image acceleration. Thank all the students who have participated in the community construction. I hope that more students who are interested in the field of image acceleration or P2P can join our community.