This topic describes how to determine whether Shared Memory Communication (SMC) acceleration is suitable for a scenario and whether the conditions for using SMC acceleration are met.
Overview
As applications in traditional data centers increasingly migrate to the cloud, network latency and throughput face new challenges. Traditional virtual network interfaces and TCP stacks cannot meet the high throughput and low latency requirements of modern data centers. With the improvement of Data Processing Unit (DPU) capabilities in cloud computing scenarios, Remote Direct Memory Access (RDMA) networks are offered to fully utilize hardware capabilities and enhance network performance. In this context, a software and hardware collaborative protocol stack that leverages underlying DPU hardware performance, supports large-scale cloud applications, and is developer-, deployment-, and maintenance-friendly, becomes a vital choice for cloud-based data center networks.
SMC, which was open-sourced by IBM to the Linux community and is enhanced in Alibaba Cloud Linux 3, meets the preceding requirements. SMC is a kernel network protocol stack that runs parallel to the TCP/IP stack but is more lightweight. SMC offloads network protocol processing, such as packet encapsulation and parsing, to underlying hardware devices, effectively reducing latency and CPU overhead. Additionally, SMC transmits data by directly reading and writing to remote shared memory and significantly improves network throughput.
SMC is categorized into SMC over RDMA (SMC-R) based on RDMA technology and SMC over Direct Memory Access (SMC-D) based on internal shared memory (ISM) technology. Alibaba Cloud Linux 3 provides SMC-R acceleration capabilities based on Elastic RDMA (eRDMA). The following description pertains to SMC-R.
In terms of design, the SMC stack is compatible with traditional network socket interfaces, and its development and usage methods are similar to those of traditional TCP applications. By intercepting the socket() system call to change the socket family and protocol parameters, existing TCP applications can seamlessly switch to the SMC stack without the need to modify or recompile the applications. For more information, see Use SMC.
When establishing a communication connection, the SMC stack initially establishes a TCP connection with the peer. During the TCP three-way handshake, a special TCP option (Kind: 254, Value: 0xE2D4C3D9) is included in the SYN packet to indicate the SMC capability and check whether the peer supports SMC. If the handshake is successful, both ends create corresponding RDMA resources and complete an additional handshake. Subsequent data traffic is transmitted through the RDMA network. If the handshake fails, both ends securely fall back to the established TCP connection, which ensures uninterrupted communication even if the peer does not support SMC.
After establishing a connection, the SMC stack creates an RDMA reliable connection (RC) (SMC link) for the first SMC connection. Traffic is transmitted over the SMC link through the RDMA network. To efficiently use the RDMA link and minimize RDMA resource overhead, subsequent SMC connections attempt to reuse this SMC link. When the reuse limit is reached, a new SMC link is created. Multiple SMC links can form an SMC link group (LGR) to provide RDMA transmission capabilities and achieve disaster recovery.
Dependencies
eRDMA capability
To use SMC-R on Alibaba Cloud Elastic Compute Service (ECS), configure an eRDMA network interface (ERI). For information about the instance types that support ERIs and how to configure an ERI, see Configure eRDMA on an enterprise-level instance.
Operating system
Alibaba Cloud Linux 3 supports SMC-R capabilities starting from kernel
5.10.134-16. For information about how to view and update the kernel, see Change the kernel version.
Scenarios
SMC is suitable for the following scenarios:
Scenarios in which you do not want to modify user-mode programs
A key advantage of SMC is enabling TCP applications to utilize RDMA technology without any modifications to accelerate network performance. However, as a kernel protocol stack, SMC incurs additional system calls and kernel-mode and user-mode data copy overheads, which results in inferior performance to native RDMA applications. SMC is ideal for scenarios in which seamless integration with existing applications is preferred.
Latency-sensitive and throughput-sensitive scenarios
SMC offloads network protocol processing to underlying hardware devices, effectively reducing latency and CPU overhead. SMC also improves network throughput by enabling direct memory access for data transmission. Therefore, SMC is suitable for scenarios that prioritize latency and throughput metrics.
Internal network scenarios
SMC requires support from both ends. This is more feasible in internal network scenarios in which SMC is easy to enable on the server and the client. The short network links in these scenarios also amplify the network benefits of SMC.
Large proportion of network resources
Services with a large proportion of network resources will see more significant improvements in network latency and throughput with SMC.
Prevent disadvantageous scenarios
SMC may perform worse than TCP or fail to achieve expected results in certain scenarios. Do not use SMC in the following scenarios.
Scenario | Description | Solution |
Short lifecycle communication | The establishment of an SMC connection involves multiple processes such as TCP handshake, SMC handshake, and RDMA resource creation (when using SMC-R), which results in poor connection performance and inferior performance to TCP in short lifecycle communication. | Use the |
Large number of burst connections | Similar to short lifecycle communication scenarios, the poor connection performance of SMC leads to queuing and waiting in the connection establishment process if a large number of burst connections are initiated, and results in connection establishment timeouts. | |
Cross-network element communication | During the establishment of an SMC connection, a TCP connection is first established with the peer. In the TCP three-way handshake, SYN/SYN-ACK packets include a special TCP Experimental Option to indicate the support for SMC.
However, some network elements incorrectly discard SYN packets with unrecognized options or unconditionally replay this option in the returned SYN-ACK packet, which may lead to handshake timeouts or incorrect judgments of the peer support for SMC and result in connection establishment timeouts or blockages. This issue occasionally occurs when Internet services are accessed. An automated check tool can be used to check whether the service link passes through abnormal network elements. For specific operations, see Automated check tool. | |
Public network communication | Similar to cross-network element communication scenarios, public network communication often involves cross-network element access, which can easily lead to connection establishment timeouts or blockages. Additionally, the conditions for using SMC on both ends in public network communication are difficult to meet, which results in a fallback to TCP. Finally, SMC is designed to accelerate internal data center networks and does not provide a security hardening mechanism. SMC is unsuitable for untrusted environments. | |
Constrained memory resources | SMC is based on shared memory communication and uses pre-allocated buffers as shared memory, which incurs memory overhead. By default, an SMC connection occupies a buffer size of |
|
Cross-zone communication | In cross-zone scenarios, the transmission latency is anywhere from hundreds of microseconds to a few milliseconds, which reduces the latency advantage brought by SMC and results in no significant benefit. On the other hand, the increase in the bandwidth-latency product of the network requires increasing the transmit and receive buffers to fully utilize the bandwidth, increasing memory resource pressure. | Use Unique Endpoint Identifiers (UEIDs) to prevent using SMC communication across zones. For more information, see Enable and configure SMC. |
Automated check tool
Alibaba Cloud Linux 3 provides the aliyunsmc-check tool for automated check of SMC-related basic environment configurations, communication links, and communication capabilities. The aliyunsmc-check tool is included in the aliyun-smc-extensions package.
Install the package
Run the following command to install the
aliyun-smc-extensionspackage:sudo yum install -y aliyun-smc-extensionsBasic environment configuration check
The basic environment configuration check verifies the correctness of the basic environment, such as the operating system kernel version and the presence of eRDMA devices. Run the following command to perform the check:
aliyunsmc-check basic_checkSample command output
Communication link check
The communication link check verifies whether network elements on a communication link affect SMC communication to determine the suitability of the link for SMC communication. Run the following command to perform the check. Replace
<url>with an actual URL.aliyunsmc-check syn_check --url <url>Sample command output
If the URL is correct and the link fails the check, the link is unsuitable for SMC communication. Use the
smc-ebpftool to disable SMC communication on the link. For more information, see Enable and configure SMC.SMC communication check
The SMC communication check verifies whether SMC can be used for normal communication between the current machine and a remote host specified by the
hostsparameter. During the check,SSHis used to log on to the remote host.SSHauthentication information is required in the command parameters.The following
SSHauthentication methods are supported:Use
keyfilefor SSH logon.Replace
<peer ip>with the IP address of the remote host.Replace
<usr>with the username of the remote host.Replace
<keyfile>with the path of the SSH private key file.
aliyunsmc-check connect_check --hosts <peer ip> --user <usr> --key <keyfile>Use a password for SSH logon.
Replace
<peer ip>with the IP address of the remote host.Replace
<usr>with the username of the remote host.Replace
<passwd>with the password of the remote host.
aliyunsmc-check connect_check --hosts <peer ip> --user <usr> --pwd <passwd>Use SSH passwordless logon.
Replace
<peer ip>with the IP address of the remote host.Replace
<usr>with the username of the remote host.
aliyunsmc-check connect_check --hosts <peer ip> --user <usr>
Sample command output
