Alibaba Cloud Linux:SMC issues

SMC does not improve application performance over TCP

SMC connections may fall back to TCP without Remote Direct Memory Access (RDMA) acceleration. Check whether a fallback occurred by running smcss -a. If the output shows TCP in the mode column, see SMC falls back to TCP for the cause code lookup.

Other common reasons SMC may not improve performance:

• The application is CPU-bound. If the application spends most of its time on computation rather than network I/O, switching to RDMA has little effect.

• RDMA header overhead. RDMA packets carry additional headers compared to TCP. For the same available bandwidth, RDMA achieves slightly lower throughput. To mitigate this, enable Jumbo Frames.

• Short-lived connections. SMC connection setup involves slow-path operations such as creating and requesting RDMA resources. For workloads dominated by short-lived connections, this overhead outweighs the benefit.

• Insufficient resources. SMC requires memory and elastic RDMA interface (ERI) resources tied to the Elastic Compute Service (ECS) instance specifications. When resources run out, SMC falls back to TCP. For resource requirements, see Enable and configure SMC.

Communication fails after SMC is enabled

After enabling SMC-R on an Alibaba Cloud Linux 3 ECS instance, some Internet-facing addresses can be pinged but not accessed. cURL requests fail while ICMP pings succeed. Disabling SMC-R resolves the issue.

This happens when remote servers replay TCP options instead of ignoring unsupported ones (as required by RFC 9293). If a server echoes back the SMC TCP option, the local end incorrectly identifies the peer as SMC-capable. The resulting handshake mismatch causes connection failures.

Run a communication link check to diagnose the problem.

To fix this, configure SMC negotiation control based on BPF policies and disable SMC on the problematic link.

SMC is not enabled after running smc_run

Running smc_run ./foo does not create SMC connections. The smcr l output shows no link groups, and smcss -a shows either no SMC connections or a one-sided TCP fallback.

smc_run uses LD_PRELOAD to inject smc-tools libraries that intercept socket(2) calls and modify socket families and protocols. This mechanism does not work for statically linked applications.

For statically linked applications, enable SMC at the kernel level instead:

sysctl net.smc.tcp2smc

For details, see Enable and configure SMC.

Ports 65500-65515 become unusable after SMC is enabled

After loading SMC modules, bind(2) calls on ports 65500 through 65515 return EADDRINUSE.

SMC-R with elastic Remote Direct Memory Access (eRDMA) reserves these 16 ports for out-of-band (OOB) connections in the net namespace where ERIs reside. Run dmesg to verify:

smc: smc: load SMC module with reserve_mode
NET: Registered protocol family 43
smc: netns <netns ID> reserved ports [65500 ~ 65515] for eRDMA OOB
smc: adding ib device erdma_0 with port count 1
smc: ib device erdma_0 port 1 has pnetid

If these ports are already in use when SMC modules load, the modules cannot use eRDMA devices.

To release the ports, unload the SMC modules. See Use SMC in Alibaba Cloud ECS.

SMC falls back to TCP with IPv6 addresses

After enabling SMC for applications that use IPv6, smcss shows TCP fallback with cause code 0x03030000 or 0x09990000.

Alibaba Cloud eRDMA devices and SMC do not support IPv6. Use one of these workarounds before enabling SMC for new connections:

Disable IPv6 for all interfaces:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Disable IPv6 for a specific interface:

sudo sysctl -w net.ipv6.conf.<NetInName>.disable_ipv6=1

Replace <NetInName> with the interface name.

Use IPv4-mapped IPv6 addresses on kernel version 5.10.134-17.3 and later.

SMC performance is lower than TCP at the PPS limit

When the network load reaches the maximum packets per second (PPS) rate defined by the ECS instance type, applications using SMC with eRDMA show lower queries per second (QPS) than those using TCP.

Check whether you have hit the PPS limit:

• TCP traffic: Run sar -n DEV 1 and check rxpck/s and txpck/s.

• SMC eRDMA traffic: Run eadm stat -d <ibdev_name> -l to check eRDMA packet rates. For details on eadm, see Use eadm to diagnose and troubleshoot issues in eRDMA.

RDMA generates more network packets per request than TCP, so the PPS limit is reached sooner. This only occurs under extreme network load such as benchmark stress tests. In production, traffic rarely hits the PPS ceiling.

If the PPS limit is the bottleneck, do not use SMC for that workload.

SMC falls back to TCP and RDMA cannot accelerate communications

After enabling SMC, smcss -a shows that the connection fell back to TCP. The connection still works, but without RDMA acceleration.

State          UID   Inode   Local Address           Peer Address            Intf Mode
ACTIVE         00000 0156721 192.168.99.21:60188     192.168.99.22:8090      0000 TCP 0x03010000
ACTIVE         00000 1202539 172.16.4.189:44780      172.16.4.190:1811       0000 SMCR

Network O&M tools show unexpected data after SMC is enabled

After enabling SMC, tools such as tcpdump, Wireshark, ss, and netstat show network traffic data that does not match expectations, or do not capture expected traffic.

SMC-R is based on RDMA. These tools analyze only TCP traffic and cannot identify RDMA packets.

Use RDMA-specific tools instead. See Monitor and check eRDMA.

SMC module is unusable on GPU-accelerated or SCC instances

The SMC module loaded on a GPU-accelerated or Super Computing Cluster (SCC) instance does not function.

These instance types have Mellanox OpenFabrics Enterprise Distribution (OFED) drivers installed. The OFED stack includes its own SMC module that auto-loads but does not work. After installing Mellanox OFED drivers, RDMA function symbols change, and the kernel SMC module fails to load with an Unknown symbol error.

SMC cannot be used on GPU-accelerated or SCC instances with Alibaba Cloud Linux 3.

Some SOL_SOCKET and SOL_TCP options do not work after enabling SMC

After replacing TCP with SMC, some setsockopt and getsockopt options at the SOL_SOCKET or SOL_TCP level cannot be configured, cannot be retrieved, or do not work as expected.

SMC uses shared buffers and a different protocol stack design than TCP. Some socket options are incompatible with this design.

The support levels are:

• Y: Fully supported. The option can be set, retrieved, and works as expected.

• M: Configurable but may not work as expected due to design differences between SMC and TCP.

• N: Not supported. Using the option causes a TCP fallback with cause code 0x03060000 or 0x03010001.