Alibaba Cloud Linux:Using Shared Memory Communication (SMC)

Prerequisites

Before you begin, ensure that you have:

• An ECS instance that supports the Elastic RDMA Interface (ERI) to get elastic Remote Direct Memory Access (eRDMA) capability. See Configure eRDMA on an enterprise-level instance.

Set up SMC

1. Load the SMC kernel modules.

sudo modprobe smc
sudo modprobe smc_diag

Run dmesg to confirm the modules loaded:

smc: smc: load SMC module with reserve_mode
NET: Registered protocol family 43
smc: netns <netns ID> reserved ports [65500 ~ 65515] for eRDMA OOB
smc: adding ib device erdma_0 with port count 1
smc:    ib device erdma_0 port 1 has pnetid

To unload SMC when no longer needed:

sudo rmmod smc_diag
sudo rmmod smc

2. Install the O&M toolsets.

sudo yum install -y smc-tools
sudo yum install -y aliyun-smc-extensions

Enable net namespace-level socket conversion

This method converts all qualifying TCP sockets created after the sysctl is set. Existing sockets are not affected.

The following figure shows the conversion process.

1. Enable transparent socket conversion for the current net namespace:

   sudo sysctl net.smc.tcp2smc=1

   The default value is 0 (disabled).

2. Start your TCP socket application normally:

   ./<foo>

   Replace <foo> with your application name. The application's TCP sockets are transparently converted to SMC sockets and handled by the SMC-R stack. See How it works for the fallback behavior.

3. (Optional) Disable conversion for the current net namespace. New TCP sockets revert to TCP, but existing converted SMC sockets are affected:

   sudo sysctl net.smc.tcp2smc=0

Enable process-level socket conversion

This method applies to a single process and its child processes using smc_run from the smc-tools toolset.

The following figure shows the conversion process.

Prefix your application's executable with smc_run:

smc_run ./<foo>

Replace <foo> with your process name. The TCP sockets created by the process are transparently converted to SMC sockets. See How it works for the fallback behavior.

Install smc-ebpf

smc-ebpf is included in smc-tools. Verify the installation:

smc-ebpf policy help

A successful response lists the available subcommands:

smc-ebpf policy help
 Usage: smc-ebpf policy COMMAND [OPTIONS]
        smc-ebpf policy load [OPTIONS]    load policy
    --init                                load policy with pre-defination config
        smc-ebpf policy stop              stop policy
        smc-ebpf policy unload            unload policy
        smc-ebpf policy init              init policy with default config
        smc-ebpf policy clear             clear all policy config
        smc-ebpf policy dump              display all policy config
        smc-ebpf policy config [OPTIONS]  config policy
        smc-ebpf policy delete [OPTIONS]  delete policy
    --ip [IPv4]                           target IPv4 address
    --port                                target port
    --mode [auto|disable|enable]          target mode
 Examples:
    smc-ebpf policy load
    #disable port 80 to use smc
    smc-ebpf policy config --port 80 --mode disable
    #delete ip xxx.xxx.x.x/24 policy
    smc-ebpf policy delete --ip xxx.xxx.x.x --mask 24

Load smc-ebpf

sudo smc-ebpf policy load

A successful load outputs:

# sudo smc-ebpf policy load
Registered smc_sock_negotiator_ops anolis_smc id xxx

If the output differs, troubleshoot as follows:

• Confirm the kernel version is ANCK 5.10.134-016 or later: uname -r

• Confirm you have permission to load BPF programs. Containers may lack the required capability — check with your environment provider.

Port policies

Port policies control whether SMC negotiation is used based on the connection's port.

Default behavior: After loading smc-ebpf, any port without a matching policy is prohibited from using SMC negotiation.

To change the default to allow SMC for unmatched ports:

sudo smc-ebpf policy config --port 0 --mode enable

To reset the default back to prohibit:

sudo smc-ebpf policy config --port 0 --mode disable

Example 1: Allow only port 80, block all others

# Allow port 80
sudo smc-ebpf policy config --port 80 --mode enable

# Verify
sudo smc-ebpf policy dump

Expected output:

# sudo smc-ebpf policy dump
[{
        "key": 80,
        "value": {
            "mode": 2,
            [Other fields that ordinary users do not need to pay attention to]
        }
    }
]

"key": 80 is the port. "mode": 2 means SMC negotiation is allowed; "mode": 0 means prohibited.

To remove the policy:

sudo smc-ebpf policy delete --port 80

Example 2: Block only port 80, allow all others

# Change default: allow SMC for unmatched ports
sudo smc-ebpf policy config --port 0 --mode enable
# Block port 80
sudo smc-ebpf policy config --port 80 --mode disable

IPv4 address policies

IPv4 address policies apply to client sockets only. They control whether the client uses SMC negotiation when connecting to a specific server IP address. Server socket behavior is unaffected.

All port policies and IPv4 address policies are evaluated with a logical AND. SMC negotiation is used only when every matching policy allows it.

Default behavior: After loading smc-ebpf, any server IP without a matching policy is allowed for SMC negotiation.

To change the default to block SMC for unmatched IPs:

sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode disable

To reset the default back to allow:

sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode enable

Example 1: Allow SMC only to 192.168.2.0/24, block all other destinations

# Block unmatched IPs
sudo smc-ebpf policy config --ip 0.0.0.0 --mask 32 --mode disable
# Allow 192.168.2.0/24
sudo smc-ebpf policy config --ip 192.168.2.0 --mask 24 --mode enable

# Verify
sudo smc-ebpf policy dump

Expected output:

# sudo smc-ebpf policy dump
key:     0.0.0.0/32           value:   "denied"
key:     192.168.2.0/24       value:   "pass"

"pass" = SMC allowed; "denied" = SMC prohibited.

To remove these policies:

sudo smc-ebpf policy delete --ip 192.168.2.0 --mask 24
sudo smc-ebpf policy delete --ip 0.0.0.0 --mask 32

Example 2: Block SMC only to 192.168.2.0/24, allow all other destinations

sudo smc-ebpf policy config --ip 192.168.2.0 --mask 24 --mode disable

No further configuration is needed. The default allows SMC for all other server IPs.

Clear all policies

sudo smc-ebpf policy clear

This deletes all configured policies and restores the default state: SMC negotiation is prohibited for all ports (port default: prohibit) and allowed for all IPv4 addresses (IP default: allow). Because port and IP policies are ANDed, all connections end up prohibited from using SMC negotiation.

Sysfs parameters

Configure EIDs

Enterprise ID (EID) is a concept in the SMCv2 protocol. Only systems sharing the same EID can communicate over SMCv2; otherwise they fall back to TCP. A system supports up to eight EIDs.

Alibaba Cloud Linux 3 eRDMA devices require the SMCv2 protocol. All Alibaba Cloud Linux 3 nodes ship with the EID SMCV2-DEFAULT-UEID, so nodes can communicate over SMCv2 without any manual EID configuration.

EID format: up to 32 characters, uppercase letters, digits, hyphens (-), and periods (.); must start with a letter or digit; no consecutive periods.

To view, add, or remove EIDs:

# View existing EIDs
smcr ueid show

# Add an EID
sudo smcr ueid add <EID>

# Remove an EID
sudo smcr ueid del <EID>

# Fetch the zone ID from instance metadata and convert to uppercase
ZONE_ID=$(curl -s -m 1 100.100.100.200/latest/meta-data/zone-id | tr "[:lower:]" "[:upper:]")

# Add the zone ID as an EID
sudo smcr ueid add $ZONE_ID

# Remove the default EID
smcr ueid | grep SMCV2-DEFAULT-UEID > /dev/null && sudo smcr ueid del SMCV2-DEFAULT-UEID

sudo systemctl start aliyunsmc-ueid

sudo systemctl enable aliyunsmc-ueid

Configure PNET IDs

When TCP traffic is converted to SMC-R, the SMC-R stack routes traffic through the Elastic RDMA Interface (ERI) associated with the Ethernet interface carrying that traffic.

This association works automatically if eRDMA is enabled directly on the Ethernet interface. If the ERI is associated with a different Ethernet interface, use a physical-network identifier (PNET ID) to link them manually.

Example: A node has eth0 and eth1. eRDMA is enabled on eth0, which is associated with erdma_0.

• TCP traffic on eth0 — SMC-R detects erdma_0 automatically. No PNET ID needed.

• TCP traffic on eth1 — SMC-R cannot find an ERI and falls back to TCP. Associate eth1 with erdma_0 via a PNET ID to enable RDMA on eth1 traffic.

PNET ID format: up to 16 characters, uppercase letters and digits only, no spaces.

To associate an Ethernet interface and an ERI with a PNET ID:

# Assign the PNET ID to the Ethernet interface
sudo smc_pnet -a <PNET ID> -I <eth_interface>

# Assign the same PNET ID to the ERI
sudo smc_pnet -a <PNET ID> -D <rdma_interface>

To view configured PNET IDs:

sudo smc_pnet

Sample output:

# sudo smc_pnet
00163E0CD751 n/a erdma_0 1
00163E0CD751 eth1 n/a 255

PNET ID 00163E0CD751 links erdma_0 and eth1. TCP traffic through eth1 is now accelerated over erdma_0 when SMC-R is enabled.

For reference on checking and enabling eRDMA on an Ethernet interface:

• View ERIs

• Create an ERI

• Change the ERI status for an existing ENI

Use PNET IDs for pods sharing host ERIs

In self-managed container environments where multiple pods share one or more host ERIs, assign the same PNET ID to both the pod's virtual Ethernet interface (vEth) and the host ERI. This lets the kernel detect the host ERI and use RDMA to accelerate pod traffic.

# Assign the PNET ID to the vEth in the pod's net namespace
sudo ip netns exec <pod netns> smc_pnet -a <PNET ID> -I eth0

# Assign the same PNET ID to the host ERI
sudo smc_pnet -a <PNET ID> -D erdma_0