Cilium High Performance Cloud Native Network

Recently, the Alibaba Cloud team introduced the new high-performance container network solution of Alibaba Cloud Container Service at the SIG Cloud-Provider-Alibaba conference and released a blog introduction. Did you know that this solution is implemented based on Cilium & eBPF. Prior to this, Google's GKE and Anthos also announced the implementation of a new container network data plane V2 solution based on Cilium+eBPF. However, Alibaba Cloud’s solution will be different. Alibaba Cloud adopts the combination of Terway IPVLAN+Cilium’s eBPF. In the following article, we will analyze the detailed implementation of Terway CNI (Alibaba Cloud’s CNI plug-in) and the test data in the blog.

Like other cloud vendors, Alibaba Cloud also provides ENI (Elastic Network Interface) products to expose the SDN (Software Defined Network) capabilities of the underlying IAAS layer. For K8S pods, cloud-native virtualized networks can be realized based on it, without the need to add another layer of virtualization to the container network to reduce performance loss and network complexity.

The IAAS layer network of cloud vendors already has the capabilities of virtualization and SDN. If the capabilities of the underlying virtualized network are directly used by Pods, performance loss will be significantly reduced.

For Alibaba Cloud, the container network model is shown in the following figure:

In order to implement this model, the CNI layer directly interacts with Alibaba Cloud's API to apply for the underlying ENI network resources required by the Pod. Alibaba Cloud has developed Terway's CNI plug-in to implement such a model. There is a detailed introduction to the internal implementation and the challenges encountered in the Alibaba Cloud official blog. Here we focus on how they use IPVLAN and eBPF to improve the performance and scalability of Kubernetes' Service and NetworkPolicy.

Use IPVLAN for better network scalability and performance

A single ENI can be exclusive to a Pod or shared among multiple Pods. When an ENI is shared by multiple Pods, some routing decisions need to be made on packets to ensure that Pod traffic is routed to its corresponding ENI. Using the shared ENI method, one ENI can virtualize 10-20 IPs, which can greatly increase the deployment density of Pods on the node, but the disadvantage is that bridges or policy routing need to be introduced to bring additional performance overhead. The specific overhead can be seen in the performance comparison later.

In order to improve the performance of shared ENI, IPVLAN is a good choice. IPVLAN can lightly virtualize ENI into multiple sub-interfaces to connect multiple Pods to a single ENI. Terway's CNI uses IPVLAN to reduce the overhead of sharing ENI, and combines Cilium to provide efficient NetworkPolicy and Service implementations in IPVLAN network mode. And submitted a pull request to Cilium official.

The following is a performance comparison of different modes, which also includes the performance advantages of the cloud-native ENI network and the overlay-based Flannel.

You don't have to choose one of the models, you can schedule the exclusive ENI for high performance as needed, and use the shared ENI mode for other Pods.

**Use eBPF to solve the scalability problem of Kubernetes Service and NetworkPolicy
**
For a long time, the standard kube-proxy implementation of Kubernetes has adopted the iptables mode. Due to the order matching of iptables, the scalability of this solution is very limited.

It can be seen that when the number of services increases to a certain threshold, the delay will increase significantly. What's more serious is that due to the different matching order of the service table items in the iptables rule chain, the delay of the first packet accessed by the service will be randomly changed.

For these reasons, Alibaba Cloud optimizes the scalability of Kubernetes based on eBPF.

How is the effect? The following is the performance comparison tested by the Alibaba Cloud team. The network performance and scalability of the eBPF-based solution are better than the iptables and IPVS modes of kube-proxy:

The link is simplified through eBPF, and the performance is significantly improved. Compared with the iptables mode, the performance is improved by 32%, and compared with the IPVS mode, the performance is improved by 62%.

Similar to Kubernetes Server, the NetworkPolicy of Kubernetes can also be optimized based on eBPF.

The "BPF-agent" in the box is the Cilium agent that runs independently of Terway CNI, and is used to provide Kubernetes Service and NetworkPolicy implementations:

We use Cilium as the BPF-agent on the node to configure the BPF rules of the container NIC, and have contributed Terway-related adaptations.

Unfortunately, Alibaba Cloud did not provide a final optimization comparison in this article. The Cilium team made a comparison blog of Cilium in IPVLAN and veth mode earlier, which can be used as a rough reference.
Summarize
We are very happy and welcome Alibaba Cloud to join and contribute to the Cilium community. If you need to know more, please refer to the following:
• Cilium Overview
• Cilium GitHub
• How Does Alibaba Cloud Build High-Performance Cloud-Native Pod Networks in Production Environments?
• What is eBPF?