Monitor Shared Memory Communications (SMC) to find and resolve issues that may affect performance - Alibaba Cloud Linux

Monitor Shared Memory Communication (SMC) health by tracking protocol stack statistics, active connections, and underlying RDMA device state. All metrics are collected with tools from the smc-tools package.

Prerequisites

Before you begin, ensure that you have:

Alibaba Cloud Linux 3 installed on your instance
smc-tools installed (the user-mode SMC maintenance toolset)

To install smc-tools, run:

sudo yum install -y smc-tools

Monitor protocol stacks

The SMC stack exposes connection, traffic, and shared memory metrics to user space via netlink. The smcr command reads and interprets these metrics.

Query SMC-R stack statistics

Run smcr stats to query statistics about the SMC over Remote Direct Memory Access (SMC-R) stack in the current net namespace:

smcr stats

Sample output:

# smcr stats
SMC-R Connections Summary
  Total connections handled          5076
  SMC connections                    5076
  Handshake errors                      0
  Avg requests per SMC conn          1977.0
  TCP fallback                          0

RX Stats
  Data transmitted (Bytes)      200705600 (200.7M)
  Total requests                  5017741
  Buffer usage (Bytes)                  0 (0)
  Buffer full                           0 (0.00%)
            8KB    16KB    32KB    64KB   128KB   256KB   512KB  >512KB
  Bufs        0       0       0       0       0  5.076K       0       0
  Reqs   5.018M       0       0       0       0       0       0       0

TX Stats
  Data transmitted (Bytes)     1194173445 (1.194G)
  Total requests                  5017640
  Buffer usage (Bytes)                  0 (0)
  Buffer full                           0 (0.00%)
  Buffer full (remote)                  0 (0.00%)
  Buffer too small                      0 (0.00%)
  Buffer too small (remote)             0 (0.00%)
            8KB    16KB    32KB    64KB   128KB   256KB   512KB  >512KB
  Bufs        0       0       0       0       0  5.076K       0       0
  Reqs   5.018M       0       0       0       0       0       0       0

Extras
  Special socket calls                  5

The following table describes the key parameters.

Parameter	Description
`Total connections handled`	Total connections handled by the SMC-R stack. Equals the sum of `SMC connections`, `Handshake errors`, and `TCP fallback`.
`SMC connections`	Connections successfully converted to SMC-R.
`Handshake errors`	Connections that failed during the handshake phase (for example, no response from the peer).
`Avg requests per SMC conn`	Average number of requests sent or received per SMC connection.
`TCP fallback`	Connections that fell back to TCP/IP after SMC negotiation failed.
`Rx/Data transmitted (Bytes)`	Total bytes received over SMC-R connections.
`Rx/Total requests`	Total requests received over SMC-R connections.
`Rx/Buffer usage (Bytes)`	Total size of Rx buffers currently in use. Unit: bytes.
`Rx/Buffer full`	Number of times Rx buffers became full. A full Rx buffer backpressures the sender and prevents new data from arriving. To reduce this count, configure your application to read from the Rx buffer sooner, or increase the Rx buffer size.
`Rx/Bufs`	Distribution of Rx buffer sizes allocated to SMC-R connections. SMC-R maintains a memory pool per link group. When a connection opens, SMC-R allocates an idle block of the right size from the pool (or creates a new one). When the connection closes, the block returns to the pool. This counter tracks total allocations—including both created and reused blocks—not the number of buffers actively consuming memory.
`Rx/Reqs`	Size distribution of requests received over SMC-R connections.
`Tx/Data transmitted (Bytes)`	Total bytes sent over SMC-R connections.
`Tx/Total requests`	Total requests sent over SMC-R connections.
`Tx/Buffer full`	Number of times Tx buffers became full because the SMC-R stack could not flush data to the link in time. If this percentage is high, increase the Tx buffer size.
`Tx/Buffer full (remote)`	Number of times the peer's Rx buffer was full, preventing the local end from sending. If this percentage is high, increase the peer's Rx buffer size.
`Tx/Buffer too small`	Number of times a send request exceeded the Tx buffer capacity. If this percentage is high, increase the Tx buffer size.
`Tx/Buffer too small (remote)`	Number of times a send request exceeded the peer's Rx buffer capacity. If this percentage is high, increase the peer's Rx buffer size.
`Tx/Bufs`	Distribution of Tx buffer sizes allocated to SMC-R connections. Follows the same pool logic as `Rx/Bufs`—tracks total allocations, not actively allocated buffers consuming memory.
`Tx/Reqs`	Size distribution of requests sent over SMC-R connections.

Query link group statistics

Note

By default, each link group contains one link and can carry up to 32 SMC-R connections. A link group manages a set of RDMA resources, including Queue Pairs (QPs), Protection Domains (PDs), and Memory Registrations (MRs).

Run smcr linkgroup to list all SMC-R link groups associated with RDMA devices accessible in the current net namespace:

smcr linkgroup

Sample output:

# smcr linkgroup
LG-ID    LG-Role  LG-Type  VLAN  #Conns  PNET-ID
00000300 SERV     SINGLE      0       0

Monitor the number of link groups (which equals the number of QPs in use) and the #Conns field:

Parameter	Description
`#Conns`	Number of SMC connections carried by the link group.

For detailed link group information, run:

smcr -d linkgroup

Sample output:

LG-ID    : 00000500
LG-Role  : CLNT
LG-Type  : SINGLE
VLAN     : 0
PNET-ID  :
Version  : 2
Peer-Rel : 1
Peer-Host:
Peer-OS  : LINUX
Direct   : Yes
EID      : SMCV2-DEFAULT-UEID
#Conns   : 32
Sndbuf   : 8388608 B
CNY      : 8388608 B

Parameter	Description
`Sndbuf`	Total memory occupied by the Tx buffer pool maintained by the link group. Unit: bytes.
`CNY`	Total memory occupied by the Rx buffer pool maintained by the link group. Unit: bytes.

Query RDMA device information

Run smcr device to list the RDMA devices used by the SMC-R stack:

smcr device

Sample output:

# smcr device
Net-Dev         IB-Dev   IB-P  IB-State  Type          Crit  #Links  PNET-ID
eth1            erdma_0     1    ACTIVE  0x107f          No       0

Parameter	Description
`Net-Dev`	Name of the Ethernet device.
`IB-Dev`	Name of the RDMA device.
`IB-P`	Port of the RDMA device.
`IB-State`	Status of the RDMA device.
`Type`	Type of the RDMA device. Alibaba Cloud elastic Remote Direct Memory Access (eRDMA) devices display `0x107f`.
`#Links`	Number of links associated with the RDMA device.
`PNET-ID`	Physical network (PNET) ID of the RDMA device. For configuration details, see Configure PNET ID parameters.

Monitor connections

smcss prints information about SMC sockets, similar to the ss command. It retrieves socket information from netlink, covering both sockets that successfully negotiated SMC and those that fell back to TCP.

By default, smcss lists connecting, closing, and connected SMC sockets in the current net namespace.

Query basic socket information

smcss

Sample output:

# smcss
State          UID   Inode   Local Address           Peer Address            Intf Mode
ACTIVE         00994 2954337 xxx.xxx.x.xx:80         xxx.xxx.x.xx:36000      0000 SMCR
ACTIVE         00994 2953297 xxx.xxx.x.xx:80         xxx.xxx.x.xx:35948      0000 TCP 0x03010000

Parameter	Description
`State`	Socket status. Valid values: `INIT` (initializing), `CLOSED` (closed), `LISTEN` (listening), `ACTIVE` (connection established), `PEERCLW1` (no further data sent to peer), `PEERCLW2` (no further data sent to or received from peer), `APPLCLW1` (no further data received from peer), `APPLCLW2` (no further data received from or sent to peer), `APPLFINCLW` (peer closed the socket), `PEERFINCLW` (socket closed locally), `PEERABORTW` (socket closed locally in an abnormal state), `PROCESSABORT` (peer closed the socket abnormally).
`Local Address`	Local IPv4 or IPv4-mapped IPv6 address and port. SMC supports IPv4 only.
`Peer Address`	Peer IPv4 or IPv4-mapped IPv6 address and port. SMC supports IPv4 only.
`Mode`	Communication mode. `SMCR` means the connection runs on the SMC-R stack. `TCP <fallback reason>` means the connection fell back to TCP/IP. A numeric code indicates the fallback reason. For the full list of fallback codes, see Fallback to TCP/IP after enabling SMC.

Query listening sockets

smcss -l

The output fields are the same as smcss.

Query SMC-R socket details

smcss -R

Sample output:

# smcss -R
State          UID   Inode   Local Address           Peer Address            Intf Mode Role IB-device       Port Linkid GID                                      Peer-GID
ACTIVE         00000 1833669 xxx.xxx.x.xx:33618      xxx.xxx.x.xx:80         0000 SMCR CLNT erdma_0         01   01     0000:0000:0000:0000:0000:xxxx:xxxx:xxxx  0000:0000:0000:0000:0000:xxxx:xxxx:xxxx

In addition to the basic smcss fields, smcss -R adds the following:

Parameter	Description
`IB-device`	Name of the RDMA device used for the connection.
`Port`	Port of the RDMA device used for the connection.
`GID`	Global ID (GID) of the RDMA device used for the connection.
`Peer-GID`	GID of the peer RDMA device.

Query all sockets

smcss -a

The output fields are the same as smcss.

Monitor devices

SMC-R uses Alibaba Cloud eRDMA devices as its underlying RDMA infrastructure. eRDMA provides user-mode tools to collect statistics about RDMA resources and device state. For details, see Monitor and diagnose eRDMA.

Integrated monitoring with smc_monitor_ex

smc_monitor_ex is a monitoring script included in smc-tools. It calls smcr and smcss internally to report SMC traffic rates, connection counts, and memory usage in a single view.

Warning

smc_monitor_ex is an experimental tool. Its interface may change in future releases.

Run smc_monitor_ex -h to see available subcommands:

# smc_monitor_ex -h
usage: smc_monitor_ex [-h] {speed,s,connection,c,memory,m,base,b} ...

SMC Monitor Tool (Experimental)

positional arguments:
  {speed,s,connection,c,memory,m,base,b}
                        commands
    speed (s)           View transfer rates
    connection (c)      View connection counts
    memory (m)          View memory usages
    base (b)            View transfer rates, connection counts, and memory
                        usages

optional arguments:
  -h, --help            show this help message and exit

Monitor traffic rates

The speed subcommand reports the SMC traffic rate and requests per second (RPS) in the current net namespace.

# smc_monitor_ex speed -h
usage: smc_monitor_ex speed [-h] [-i INTERVAL] [-r] [-m {smcr,smcd,smc}]

optional arguments:
  -h, --help            show this help message and exit
  -i INTERVAL, --interval INTERVAL
                        Interval in seconds to display transfer rates.
  -r, --raw             Display rates in B/s without converting units.
  -m {smcr,smcd,smc}, --mode {smcr,smcd,smc}
                        Mode to check, either 'smc', 'smcr' or 'smcd', default
                        is 'smc'

For example, to query SMC-R traffic rate and RPS every second:

# smc_monitor_ex speed -m smcr -i 1
Date                  Mode            Rx Rate            Rx Rps           Tx Rate            Tx Rps
2025-02-21 14:01:48   smcr            0.0 B/s            0.0 /s           0.0 B/s            0.0 /s
Date                  Mode            Rx Rate            Rx Rps           Tx Rate            Tx Rps
2025-02-21 14:01:49   smcr            0.0 B/s            0.0 /s           0.0 B/s            0.0 /s
Date                  Mode            Rx Rate            Rx Rps           Tx Rate            Tx Rps
2025-02-21 14:01:50   smcr            0.0 B/s            0.0 /s           0.0 B/s            0.0 /s

Monitor connection counts

The connection subcommand reports the number of active SMC connections and TCP fallback connections in the current net namespace.

# smc_monitor_ex connection -h
usage: smc_monitor_ex connection [-h] [-i INTERVAL]
                                 [-m {smc,smcr,smcd,fallback,all}]

optional arguments:
  -h, --help            show this help message and exit
  -i INTERVAL, --interval INTERVAL
                        Interval in seconds to display connections.
  -m {smc,smcr,smcd,fallback,all}, --mode {smc,smcr,smcd,fallback,all}
                        Mode to check, either 'all', 'smc', 'smcr', 'smcd' or
                        'fallback', default is 'all'

For example, to query the number of SMC-R connections every second:

# smc_monitor_ex connection -m smcr -i 1
Date                  Mode                 #Conn
2025-02-21 14:06:47   smcr                     0
Date                  Mode                 #Conn
2025-02-21 14:06:48   smcr                     0
Date                  Mode                 #Conn
2025-02-21 14:06:49   smcr                     0

Monitor memory usage

The memory subcommand reports the total size of ring buffers used by SMC connections in the current net namespace.

# smc_monitor_ex memory -h
usage: smc_monitor_ex memory [-h] [-i INTERVAL] [-r] [-m {smcr,smcd,smc}]

optional arguments:
  -h, --help            show this help message and exit
  -i INTERVAL, --interval INTERVAL
                        Interval in seconds to display ringbuf usages.
  -r, --raw             Display memory usages in bytes without converting
                        units.
  -m {smcr,smcd,smc}, --mode {smcr,smcd,smc}
                        Mode to check, either 'smcr', 'smcd' or 'smc', default
                        is 'smc'

For example, to query SMC-R ring buffer usage every second:

# smc_monitor_ex memory -m smcr -i 1
Date                  Mode            Rx Bufs           Tx Bufs
2025-01-06 15:14:20   smcr          512.00 KB         512.00 KB
Date                  Mode            Rx Bufs           Tx Bufs
2025-01-06 15:14:21   smcr          512.00 KB         512.00 KB

View all statistics at once

The base subcommand combines traffic rates, connection counts, and memory usage into a single output.

For example, to query all SMC-R statistics every second:

# smc_monitor_ex base -m smcr -i 1
Date                  Mode             Rx Rate      Rx Rps        Tx Rate      Tx Rps     #Conn     Rx Bufs     Tx Bufs
2025-01-06 15:17:23   smcr           1.81 GB/s   21.66 K/s          0 B/s        0 /s         2   512.00 KB   512.00 KB
Date                  Mode             Rx Rate      Rx Rps        Tx Rate      Tx Rps     #Conn     Rx Bufs     Tx Bufs
2025-01-06 15:17:24   smcr           1.82 GB/s   21.81 K/s          0 B/s        0 /s         2   512.00 KB   512.00 KB