This topic explains how to enable Shared Memory Communication (SMC), define its acceleration scope, and configure its interfaces for optimal performance.
Use SMC on Alibaba Cloud ECS
Follow these steps to use SMC-R.
Prerequisites
-
Create an Elastic Compute Service (ECS) Instance that supports the Elastic RoCE Infrastructure (ERI) feature.
To use SMC-R on Alibaba Cloud, you need an Elastic Compute Service (ECS) Instance that has Elastic Remote Direct Memory Access (eRDMA) capability. For more information, see Enable eRDMA on an enterprise-level instance.
Important-
Alibaba Cloud eRDMA devices and SMC do not currently support IPv6 addresses. If your application uses an IPv6 address, SMC falls back to TCP. For more information, see SMC falls back to TCP with IPv6 addresses.
-
Starting from
ANCK 5.10.134-17.3, SMC supportsIPv4-mapped IPv6addresses.
-
-
Load the
smcandsmc_diagKernel Modules.sudo modprobe smc sudo modprobe smc_diagRun the
dmesgcommand to view kernel log messages. If you see output similar to the following, the modules are loaded successfully.smc: smc: load SMC module with reserve_mode NET: Registered protocol family 43 smc: netns <netns ID> reserved ports [65500 ~ 65515] for eRDMA OOB smc: adding ib device erdma_0 with port count 1 smc: ib device erdma_0 port 1 has pnetid(Optional) To unload the SMC modules if you no longer need them, run the following commands:
sudo rmmod smc_diag sudo rmmod smc -
Run the following commands to install the
smc-toolsandaliyun-smc-extensionsutilities.sudo yum install -y smc-tools sudo yum install -y aliyun-smc-extensions
Run TCP applications on the SMC stack
Alibaba Cloud Linux 3 SMC-R supports transparent socket protocol conversion at two levels.
Net namespace-level socket conversion
Alibaba Cloud Linux 3 provides a feature to transparently convert sockets at the net namespace level. You can use the sysctl net.smc.tcp2smc switch to convert all TCP sockets in a net namespace to SMC sockets if they meet the following conditions:
-
The
familyis AF_INET. -
The
typeis SOCK_STREAM. -
The
protocolis IPPROTO_IP(0) or IPPROTO_TCP(6).
The following figure shows the conversion process.
Follow these steps to enable transparent conversion at the net namespace level.
-
Run the following command to enable the global
net.smc.tcp2smcswitch for thenet namespace.Newly created TCP sockets are converted to SMC sockets. Existing TCP sockets are not affected.
sudo sysctl net.smc.tcp2smc=1By default,
sysctl net.smc.tcp2smcis set to0, which means the feature is disabled. -
Run any TCP socket application in the current
net namespace.Replace
<foo>in the example with your application name../<foo>The TCP sockets created by the
fooapplication are transparently replaced with SMC sockets, and the SMC-R Protocol Stack handles the application's network behavior. As described in the Overview, if the peer also supports the SMC-R protocol and the negotiation is successful, both ends transfer data over the RDMA network. Otherwise, communication safely falls back to the TCP network. -
(Optional) Run the following command to disable the global replacement switch for the
net namespace. Newly created TCP sockets are no longer converted. Existing SMC sockets are not affected.sudo sysctl net.smc.tcp2smc=0
Process-level socket conversion
Alibaba Cloud Linux 3 also provides process-level transparent protocol conversion, which is enabled using the smc-tools utility.
When you run an application with the smc_run script, it uses the LD_PRELOAD environment variable to prioritize loading the libsmc-preload.so dynamic library. The libsmc-preload.so library converts TCP sockets that meet the following conditions in the application and its child processes into SMC sockets:
-
The
familyis AF_INET. -
The
typeis SOCK_STREAM. -
The
protocolis IPPROTO_IP(0) or IPPROTO_TCP(6).
smc_run uses LD_PRELOAD to intercept the socket() system call in glibc. Therefore, it does not work for applications that are statically linked or do not use glibc.
The following figure shows the replacement process.
Follow these steps to enable transparent conversion at the process level.
Run the following command, adding the smc_run prefix before the command for your foo application.
Replace <foo> in the example with your process name.
smc_run ./<foo>
The TCP sockets created by the foo application are transparently replaced with SMC sockets, and the SMC-R Protocol Stack handles the application's network behavior. As described in the Overview, if the peer also supports the SMC-R protocol and the negotiation is successful, both ends transfer data over the RDMA network. Otherwise, communication safely falls back to the TCP network.
SMC negotiation control with BPF
In practice, enabling SMC at the net namespace or process level can sometimes be too coarse-grained. For example, a server might have multiple listening ports in a net namespace. You may want to use SMC only for connections on ports that require performance acceleration, while connections to other ports, such as management ports, should safely fall back to TCP.
To address this, Alibaba Cloud Linux 3 supports using BPF to control SMC negotiation for connections after transparent conversion is enabled. The typical process is as follows:
-
Enable and configure BPF policies to set up granular control for SMC negotiation.
-
Enable SMC at the
net namespaceor process level. For more information, see Run TCP applications on the SMC stack.
As described in the Overview, communicating parties use a special TCP option during the TCP handshake to declare and confirm mutual SMC-R support. If the negotiation succeeds, subsequent network transmission between the two parties is based on the RDMA network. Otherwise, the connection safely falls back to the TCP network.
By default, an SMC socket always initiates and responds to this special TCP option. However, you can use BPF to gain more granular control over SMC enablement by applying policies based on ports or IPv4 addresses.
Alibaba Cloud Linux 3 provides the smc-ebpf tool (part of smc-tools) to configure these BPF policies.
Run the following command to check if smc-ebpf is installed successfully.
smc-ebpf policy help
If the following output is displayed, smc-ebpf is installed successfully.
smc-ebpf policy help
Usage: smc-ebpf policy COMMAND [OPTIONS]
smc-ebpf policy load [OPTIONS] load policy
--init load policy with pre-defination config
smc-ebpf policy stop stop policy
smc-ebpf policy unload unload policy
smc-ebpf policy init init policy with default config
smc-ebpf policy clear clear all policy config
smc-ebpf policy dump display all policy config
smc-ebpf policy config [OPTIONS] config policy
smc-ebpf policy delete [OPTIONS] delete policy
--ip [IPv4] target IPv4 address
--port target port
--mode [auto|disable|enable] target mode
Examples:
smc-ebpf policy load
#disable port 80 to use smc
smc-ebpf policy config --port 80 --mode disable
#delete ip xxx.xxx.x.x/24 policy
smc-ebpf policy delete --ip xxx.xxx.x.x --mask 24
Load smc-ebpf
Run the following command to load smc-ebpf.
sudo smc-ebpf policy load
-
smc-ebpfdoes not affect connections established before it is loaded. -
The policies configured by
smc-ebpfare system-wide and cannot be configured for specific net namespaces (containers). -
The
smc-ebpffeatures are still being standardized in the upstream community. The interfaces may change in the future and are currently considered experimental.
-
The following output indicates that the command was successful:
# sudo smc-ebpf policy load Registered smc_sock_negotiator_ops anolis_smc id xxx -
Otherwise, you cannot use this feature in the current environment. Possible reasons include:
-
Confirm that the operating system Kernel Version is
ANCK 5.10.134-016or later. You can run theuname -rcommand to check the Kernel Version. -
Confirm that you have permission to load BPF programs. A common issue is that the pod container you are using does not have the capability to load BPF programs, or your user privilege level is insufficient. You can consult your environment provider for more information.
-
Default behavior of port policies
By default, after smc-ebpf is loaded, it denies SMC negotiation for any port that does not match a policy.
For example:
-
If
smc-ebpfis loaded without any port policies configured, SMC negotiation is denied for all ports. -
If
smc-ebpfis loaded and only one policy is configured to allow SMC negotiation on port 80, SMC negotiation is denied for any connection using port 8080 because no policy matches port 8080.
You can change this default behavior by running the following command to set port 0 to enable. With this setting, if no port policy matches a target port, SMC negotiation is allowed for that port.
sudo smc-ebpf policy config --port 0 --mode enable
Similarly, you can revert to the original default behavior by setting port 0 to disable. This will again deny SMC negotiation for any port that does not have a matching policy.
sudo smc-ebpf policy config --port 0 --mode disable
Port-based policies
In addition to the default behavior, you can add policies for specific ports.
For example:
-
Example 1: Allow SMC negotiation only on port 80, and deny it for all other ports.
Run the following command to add a policy that allows SMC negotiation on port 80.
sudo smc-ebpf policy config --port 80 --mode enableRun the following command to view the port policy configuration.
sudo smc-ebpf policy dumpExample output:
-
"key": 80indicates that the policy applies to port 80. -
"mode"indicates whether SMC negotiation is allowed or denied.2means allowed, and0means denied.
# sudo smc-ebpf policy dump [{ "key": 80, "value": { "mode": 2, [Other fields can be ignored by general users] } } ]If this policy is no longer needed, run the following command to delete the policy for port 80.
sudo smc-ebpf policy delete --port 80Run the
dumpcommand again. The following output indicates that the configuration has been successfully deleted.# sudo smc-ebpf policy dump [] -
-
Example 2: Deny SMC negotiation only on port 80, and allow it for all other ports.
Run the following commands to change the default behavior to allow SMC negotiation for unmatched ports, and then add a policy to specifically deny SMC negotiation for port 80.
sudo smc-ebpf policy config --port 0 --mode enable sudo smc-ebpf policy config --port 80 --mode disableAfter this, SMC negotiation is denied only for port 80. All other ports that do not match a specific policy are allowed to use SMC negotiation.
Default behavior of IPv4 address policies
Unlike port policies, IPv4 address policies apply only to client sockets. They control whether a client uses SMC negotiation based on the server's IPv4 address when establishing a connection.
All configured port and IPv4 address policies are combined with AND logic. SMC negotiation is used only if all matching policies allow it. If any matching policy denies SMC negotiation, it will not be used.
By default, after smc-ebpf is loaded, it allows SMC negotiation for any server IP that does not match an IPv4 address policy.
For example:
-
If
smc-ebpfis loaded without any IPv4 address policies configured, client sockets are allowed to use SMC negotiation when connecting to any server IP. -
If
smc-ebpfis loaded and only one policy is configured to deny SMC negotiation for the192.168.1.0/24network, a connection to192.168.3.11will be allowed to use SMC negotiation because no policy matches192.168.3.11.
You can change this default behavior by running the following command to set 0.0.0.0/32 to disable. With this setting, if no IPv4 address policy matches a target server IP, SMC negotiation is denied for that server IP.
sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode disable
Similarly, you can restore the default behavior by running the following command. This will again allow SMC negotiation for any server IP that does not have a matching policy.
sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode enable
IPv4 address-based policies
In addition to the default behavior, you can configure policies for specific IPv4 addresses to control whether a client socket uses SMC negotiation when connecting to a specific server IP.
The configured IPv4 address filtering rules only control SMC negotiation for client sockets connecting to a server; they do not affect server sockets.
For example:
-
Example 1: Allow a client to use SMC negotiation only when connecting to servers in the 192.168.2.0/24 network. Deny it for all other server addresses.
Run the following commands to change the default behavior to deny SMC negotiation for unmatched server IPs, and then add a policy to allow it for the 192.168.2.0/24 network.
sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode disable sudo smc-ebpf policy config --ip 192.168.2.0 --mask 24 --mode enableRun the following command to view the IPv4 policy configuration.
sudo smc-ebpf policy dumpExample output:
-
key: The target IPv4 address for the policy. -
value: Indicates whether SMC negotiation is allowed or denied.passmeans allowed, anddeniedmeans denied.
# sudo smc-ebpf policy dump key: 0.0.0.0/32 value: "denied" key: 192.168.2.0/24 value: "pass"If these policies are no longer needed, run the following commands to delete the policies for
0.0.0.0/32and192.168.2.0/24.sudo smc-ebpf policy delete --ip 192.168.2.0 --mask 24 sudo smc-ebpf policy delete --ip 0.0.0.0 --mask 32 -
-
Example 2: Deny SMC negotiation for a client only when connecting to the
192.168.2.0/24network. Allow it for all other server addresses.Run the following command to add a policy that disables SMC negotiation for the
192.168.2.0/24server address range.sudo smc-ebpf policy config --ip 192.168.2.0 --mask 24 --mode disableNo other action is required. By default, after
smc-ebpfis loaded, SMC negotiation is allowed for any server IP that does not match a policy. Therefore, SMC negotiation is allowed for all server addresses except those in the192.168.2.0/24range.
Clear policies
Run the following command to clear all configured policies.
sudo smc-ebpf policy clear
Running this command deletes all configurations. The behavior reverts to the default state: SMC negotiation is denied for all ports, but allowed for all IPv4 addresses. Due to the AND logic between the two policy types, this prevents all connections from using SMC negotiation.
Use SMC on Alibaba Cloud ACK
In Container Service for Kubernetes (ACK), you can enable SMC through the ACK eRDMA Controller component. The eRDMA Controller manages eRDMA network cards, scheduling, and network capabilities for Pods. For more information, see Use SMC-R to transparently accelerate application networks.
Similar to using SMC on Alibaba Cloud ECS, when using SMC on Alibaba Cloud ACK, you can also configure granular, BPF-based SMC enablement policies on the Node.
Parameters
The SMC Protocol Stack provides configuration interfaces through sysfs and user-space tools like smc-tools. The following sections describe the configurable features of SMC.
Sysfs parameters
|
Kernel parameter |
Description |
Kernel version requirements |
Recommendations |
|
net.smc.autocorking_size |
SMC-R provides the autocork feature, similar to TCP autocork. It coalesces multiple small data packets into a larger packet for a single transmission. This improves throughput for small-packet workloads without affecting ping-pong latency. The autocorking_size parameter sets the upper limit for the size of aggregated packets. Default value: 65535. Valid values: 0 to 4294967295. A value of 0 disables the feature. |
Kernel |
For small-packet workloads that prioritize bandwidth, adjust this parameter to an appropriate value to achieve optimal throughput. For |
|
net.smc.autosplit_size |
SMC provides an autosplit feature that splits a large data packet into multiple smaller packets for batch transmission. This improves latency performance in large-packet scenarios. When a packet size exceeds 1.3 times the autosplit_size, it is split. Default value: 131072. Valid values: 32768 to 536870912. |
Kernel versions:
|
In large-packet, latency-sensitive scenarios, you can adjust this parameter to achieve optimal latency. |
|
net.smc.experiment_vendor_options |
Alibaba Cloud experimental feature options. Default value: 4294967295 (0xFFFFFFFF). |
Kernel |
We recommend that you do not modify this parameter. |
|
net.smc.global_mem |
Controls the memory watermark for SMC across the entire system. When the total size of send and receive Buffers maintained by the SMC Protocol Stack reaches Default value: [25% of total system memory, 50% of total system memory, 75% of total system memory]. |
Kernel versions:
|
You can configure |
|
net.smc.limit_smc_hs |
Controls whether to proactively fall back to TCP when connection establishment pressure is high. Default value: 1. Valid values:
|
Kernel versions:
|
We recommend setting this to 1 (enabled). In specific cases where you do not want the SMC protocol stack to automatically fall back based on connection pressure, you can set it to 0. |
|
net.smc.mem |
Controls the memory watermark for SMC in the current Default value: [25% of total system memory, 50% of total system memory, 75% of total system memory]. |
Kernel versions:
|
You can configure |
|
net.smc.rmem |
The default receive Buffer size for an SMC socket. This value is used when the receive Buffer size ( Default value: 262144. Valid values:
|
Kernel |
For more information, see SMC monitoring. |
|
net.smc.wmem |
The default send Buffer size for an SMC socket. This value is used when the send Buffer size ( Default value: 262144. Valid values:
|
Kernel |
For more information, see SMC monitoring. |
|
net.smc.smcr_buf_type |
The memory type for SMC-R send and receive Buffers. Physically contiguous memory provides better performance, but it is often difficult to allocate, which can cause Buffer sizes to be smaller than expected. In contrast, virtually contiguous memory is easier to allocate but provides slightly lower performance. Changes to this value take effect on SMC connections carried by newly created LGRs. Existing LGRs are not affected. Default value: 2. Valid values:
|
Kernel |
We recommend that you do not modify this parameter. |
|
net.smc.smcr_max_conns_per_lgr |
The maximum number of SMC connections that an LGR can carry in SMC-R. Default value: 32. Valid values:
|
Kernel |
Modify with caution.
|
|
net.smc.smcr_max_links_per_lgr |
The number of RDMA RC connections (SMC Links) that an LGR contains in SMC-R. Default value: 1. Valid values: 1 to 2. |
Kernel |
We recommend that you do not modify this parameter. |
|
net.smc.smcr_testlink_time |
The heartbeat packet interval (in seconds) for an RDMA RC connection (SMC Link) in SMC-R. When there is no data transmission on the SMC Link, 16 bytes of data are sent every Default value: 30. Valid values: 0 to 2147483647. A value of 0 disables the heartbeat check. |
Kernel |
We recommend that you do not modify this parameter. |
|
net.smc.tcp2smc |
Enables or disables transparent TCP-to-SMC conversion within the current Default value: 0. Valid values:
|
Kernel |
|
EID parameters
Introduced in the SMCv2 protocol, the Enterprise ID (EID) ensures that only systems with a matching EID can communicate via SMCv2; otherwise, they fall back to TCP. A system can have up to eight EIDs.
On Alibaba Cloud Linux 3, eRDMA devices can only be used with the SMCv2 protocol. Alibaba Cloud Linux 3 systems are initially configured with the EID SMCV2-DEFAULT-UEID. Therefore, by default, all Alibaba Cloud Linux 3 nodes can communicate with each other using SMCv2 with eRDMA without any manual configuration.
If you need to control the communication scope by modifying the EID in special circumstances, follow these steps.
-
View existing EIDs.
smcr ueid show -
Add a new EID.
An
EIDcan contain up to 32 characters, including uppercase letters (A-Z), numbers (0-9), hyphens (-), and periods (.). The first character must be a letter or a number, and periods (.) cannot be used consecutively.sudo smcr ueid add <EID> -
Delete an existing EID.
sudo smcr ueid del <EID>
Use case: Avoid cross-AZ SMC-R communication with EIDs
For optimal SMC-R acceleration, we recommend using it for communication within the same Availability Zone and using TCP for cross-Availability Zone communication. By adding the Availability Zone ID as an EID, you can achieve this behavior automatically. Follow these steps to configure it.
-
Method 1: Configure the EID step by step
-
Run the following command to get the Availability Zone ID from the Alibaba Cloud Elastic Compute Service (ECS) Instance Metadata. For more information, see Instance Metadata.
ZONE_ID=$(curl -s -m 1 100.100.100.200/latest/meta-data/zone-id | tr "[:lower:]" "[:upper:]") -
Run the following command to add the Availability Zone ID as an EID.
sudo smcr ueid add $ZONE_ID -
Run the following command to delete the default
SMCV2-DEFAULT-UEID.smcr ueid | grep SMCV2-DEFAULT-UEID > /dev/null && sudo smcr ueid del SMCV2-DEFAULT-UEID
-
-
Method 2: Use the
aliyunsmc-ueidservice from thealiyun-smc-extensionstool for one-click configuration-
Run the following command to start the
aliyunsmc-ueidservice. This service automatically adds the Availability Zone ID as an EID and removes the default EID.sudo systemctl start aliyunsmc-ueid -
(Optional) Run the following command to enable the
aliyunsmc-ueidservice to start on boot. This is necessary because the EID configuration is not persistent and resets on OS restart.sudo systemctl enable aliyunsmc-ueid
-
PNET ID parameters
TCP traffic flows through an Elastic Network Interface (ENI). When this traffic is transparently converted to SMC-R, the SMC-R Protocol Stack uses the RDMA network interface (ERI) associated with that ENI.
This association can be established in two ways.
-
The ERI is obtained by directly enabling the eRDMA capability on the target ENI. In this case, the SMC-R protocol stack automatically associates this ENI with the ERI.
-
To check if an ENI has the eRDMA capability enabled, see View ERIs.
-
To enable the eRDMA capability when creating an ENI, see Create an ERI.
-
To add the eRDMA capability to an existing ENI, see Change the status of the ERI feature for an existing ENI.
In this scenario, the SMC-R protocol stack can automatically identify and use the ERI associated with the ENI, requiring no additional operations.
-
-
The ERI is obtained by enabling the eRDMA capability on a different ENI. In this case, you must use a PNET ID to associate the ERI with the target ENI.
Consider the following scenario:
A Host has two Ethernet interfaces,
eth0andeth1. Onlyeth0has an enabled RDMA interface,erdma_0.If TCP traffic is sent and received through
eth0, switching to SMC-R allows the system to automatically find the associatederdma_0and use the RDMA network for communication. This is the automatic association scenario described above.However, if TCP traffic is sent and received through
eth1, switching to SMC-R fails to find theerdma_0interface, causing a fallback to TCP. To resolve this, associateeth1anderdma_0with the same PNET ID, which instructs SMC-R to useerdma_0foreth1's traffic.Follow these steps to associate an ENI and an ERI by using a PNET ID:
-
Run the following commands to configure the same PNET ID for the target ENI and ERI:
-
Configure a PNET ID for the ENI.
sudo smc_pnet -a <PNET ID> -I <eth_interface> -
Configure a PNET ID for the eRDMA interface (ERI).
sudo smc_pnet -a <PNET ID> -D <rdma_interface>
A PNET ID can be up to 16 uppercase alphanumeric characters without spaces.
-
-
Run the following command to check the configured PNET ID parameters.
# sudo smc_pnetExample output:
# sudo smc_pnet 00163E0CD751 n/a erdma_0 1 00163E0CD751 eth1 n/a 255In this example,
erdma_0is bound to PNET ID 00163E0CD751, andeth1is also bound to PNET ID 00163E0CD751. This allows TCP communication that originally used theeth1interface to use theerdma_0interface after enabling SMC-R.
-
Use case: Associate pod veth with host eRDMA
In scenarios where multiple pods on a single Host share an ERI, you can accelerate a pod's veth interface traffic using RDMA. To do this, assign the same PNET ID to the pod's veth Ethernet interface and the Host's eRDMA interface. This allows the kernel Protocol Stack inside the container to correctly find the Host's eRDMA interface when SMC-R is enabled.
Follow these steps to configure it.
-
In the container's
net namespace, configure a PNET ID for the container's Ethernet interface (for example,eth0).sudo ip netns exec <pod netns> smc_pnet -a <PNET ID> -I eth0 -
In the Host's
net namespace, configure the same PNET ID for the eRDMA interface (for example,erdma_0).sudo smc_pnet -a <PNET ID> -D erdma_0
Use case: Use SMC with Redis on an ECS instance
-
Create two Elastic Compute Service (ECS) Instances. One will serve as the Redis client and the other as the Redis server. For more information about the parameters, see Custom launch ECS instances.
-
Load the
smcandsmc_diagKernel Modules.sudo modprobe smc sudo modprobe smc_diag -
Run the following command on both instances to install Redis.
sudo yum install redis -y -
Run the following command on both instances to configure EIDs to avoid cross-Availability Zone SMC-R communication.
sudo systemctl start aliyunsmc-ueid -
Run the following commands on both instances to configure port-based BPF policies. This allows SMC negotiation only for connections where the server IP address is within the
vswitchCIDR block and the service port is6379.-
Run the following command to load
smc-ebpf.sudo smc-ebpf policy load -
Configure a port-based policy to allow SMC negotiation only for port
6379.sudo smc-ebpf policy config --port 6379 --mode enable -
Configure an IPv4 address-based policy to allow SMC negotiation only for server IPs within the
vswitchCIDR block.sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode disable cidr=$(curl -s -m 1 100.100.100.200/latest/meta-data/vswitch-cidr-block) sudo smc-ebpf policy config --ip \ $(echo ${cidr} | awk -F'/' '{print $1}') --mask \ $(echo ${cidr} | awk -F'/' '{print $2}') --mode enable
-
-
Run the following command on both instances to configure transparent protocol conversion at the
net namespacelevel.sudo sysctl -w net.smc.tcp2smc=1 -
On the Redis server instance, run the following command to start the Redis service.
Replace
<IP>with the private IP address of the server instance's primary Elastic Network Interface (ENI).redis-server --bind <IP> --port 6379 --protected-mode no --save -
On the Redis client instance, connect to or test the Redis server.
-
Run the following command to connect to the Redis server.
redis-cli -h <IP> -p 6379 -
Run the following command to perform a benchmark test by using
redis-benchmark.redis-benchmark -h <IP> -p 6379 -n 1000000 -t set -c 100
-