Achieving 57% Performance Increase for TCP with SMC-R

By SIG for High Performance Network

Transmission Control Protocol (TCP), as the most widely used network protocol, is commonly applied in mobile communication, data center, and other scenarios. For data centers, Shared Memory Communications over RDMA (SMC-R) of high-performance network protocol is implemented through elastic RDMA, and the application TCP protocol is transparently replaced to realize the transparent acceleration of application network.

In this blog, we'll discuss the origin of SMC-R and its significance in enhancing the performance of TCP applications. If you'd like to learn more about SMC-R, you can check out the following blogs: "Transparently Improve TCP Application Network Performance on the Cloud" and "SMC-R: A Hybrid Solution of TCP and RDMA".

Why Do We Need a New Kernel Network Protocol Stack?

The Linux kernel network protocol stack does not have a silver bullet. The current Linux network protocol stack is implemented under the tradeoff of performance (throughput and CPU usage), latency, and generality.

In real-world scenarios, we may need a high-performance but not common user-mode protocol stack, or a common, higher-performance and lower-latency solution. However, solutions based on traditional ethernet network interface controllers are difficult to be significantly improved, but benefit more from hardware, such as 100G/400G networks. In view of this, we consider whether we can provide TCP compatible behavior and socket interfaces based on other high-performance networks to provide better performance.

Network Communication Based on Shared Memory

Before talking about inter-host communication, let's focus on the stand-alone dimension. How can Inter-Process Communication (IPC) be implemented in the intra-host dimension? Here are several common IPC methods:

No doubt, shared memory is the fastest IPC method, but it lacks OS-level unified implementation and an interface. In most cases, the method is provided in the library of a specific programming language

Here, we break down the shared memory IPC process in the intra-host dimension:

The sender writes data to a pre-allocated memory area.
The sender notifies the receiver and updates the offset of the newly written memory.
The receiver reads the data according to the newly updated offset.
The receiver updates and reads the offset of the memory.

If there is a technology that can "move" memory between two machines, then we can extend this high-performance IPC solution from the stand-alone dimension to different hosts. Obviously, Remote Direct Memory Access (RDMA) can help us efficiently move memory.

Compared with the shared memory communication process of a single machine, the RDMA-based process is as follows:

The sender writes data to a pre-allocated memory area of the machine.
The sender writes the memory to the same location of the memory area maintained by the receiver through RDMA;
The sender notifies the receiver through RDMA and updates the offset of the newly written memory;
The receiver reads the data according to the newly updated offset;
The receiver updates and reads the offset of the memory through RDMA.

Based on the shared memory model of RDMA, SMC-R came into being. SMC-R is short for Shared Memory Communication over RDMA.

Let's take a look at how SMC-R accelerates TCP applications.

SMC-R is a hybrid protocol. It uses TCP to realize information interaction during connection establishment, and uses RDMA network to realize high-performance data transmission on the data path. At the same time, once the RDMA connection fails to be established, it can fallback to TCP, providing minimum communication capability over TCP. In addition, SMC-R with multiple RNICs can implement the fault migration of runtime to ensure runtime reliability.

RDMA provides the verbs interface for applications. Based on shared memory model, SMC-R offers a set of kernel interfaces that are fully compatible with TCP sockets. It can use LD_PRELOAD to transparently replace TCP sockets with SMC sockets according to replacing rules based on eBPF, and then to implement transparent replacement and acceleration.

Based on SMC-R transparent replacement, we tested several application scenarios. Among them, Redis improves its performance by up to 57%. Redis can enjoy the performance acceleration brought by SMC-R without any modification.

Use SMC-R to Accelerate Applications

To realize transparent replacement and acceleration of TCP applications, you can use the following three solutions:

Using LD_PRELOAD, the solution is to replace the SOCK_STREAM protocol created by the socket in the dynamically linked binary file with the AF_SMC protocol, thus realizing the transparent replacement of TCP with the SMC protocol.
The net namespace-level sysctl can be used to replace all TCP connections at the latitude of the network namespace such as containers.
The eBPF rules, such as quintuples and process IDs, are used to dynamically match connections to be replaced.

SMC-R in OpenAnolis

In OpenAnolis, we are also continuously enhancing and optimizing SMC, including performance, usage scenarios, stability and transparent replacement. SMC-R has participated in OpenAnolis for over six months, and contributed over 60 patches to the Linux upstream community.

You are welcome to communicate and exchange ideas in OpenAnolis. You can find relevant information from the links below.

Community

Achieving 57% Performance Increase for TCP with SMC-R

Why Do We Need a New Kernel Network Protocol Stack?

Network Communication Based on Shared Memory

Use SMC-R to Accelerate Applications

SMC-R in OpenAnolis

Related Links

Previous technical articles:

Read previous post:

Read next post:

OpenAnolis

You may also like

Comments

OpenAnolis

Related Products

Networking Overview

Remote Rendering Solution

Edge Network Acceleration

Accelerated Global Networking Solution for Distance Learning