Alibaba Cloud Linux:SMC diagnostics

Prerequisites

smc-tools, which is a user-mode SMC maintenance toolset provided by Alibaba Cloud Linux 3, is installed.

If the smc-tools toolset is not installed, run the following command to install the toolset:

sudo yum install -y smc-tools

Fallback diagnostics

The SMC stack automatically negotiates with the peer about whether to use SMC during connection establishment. If the conditions for using SMC are not met, the stack securely falls back to TCP. You can run the smcss command to query the fallback reason for each SMC connection. For more information, see Monitor connections.

Packet capture diagnostics

The SMC stack uses Remote Direct Memory Access (RDMA) and other Direct Memory Access (DMA) technologies for data transmission to bypass the traditional network protocol stack. In this case, traditional sniffing points cannot be directly used to capture data packets and control packets.

Starting from version ANCK 5.10.134-18, the Alibaba Cloud Linux 3 kernel provides a solution for capturing data packets and control packets on the SMC stack:

• When receiving or sending connection control messages and data blocks in shared memory, the SMC kernel stack constructs a UDP socket buffer (skb) that points to the target data memory and sends the skb to a specific virtual network device (dummy device). The skb is silently consumed by the virtual network device and does not enter the network. During this process, the skb passes through the existing kernel sniffing points, as shown in the following diagram with dev_queue_xmit_nit(). This allows tcpdump and libpcap to capture the data packets and control packets sent or received by the SMC stack on the virtual network device.

  

• The TCP packets in the SMC handshake process are still captured on the original Ethernet network interface controller (NIC).

The following table describes the methods for capturing packets during the stages shown in the preceding diagram.

smc-tools provides the packet capture tool smcdump-ex based on this solution, which is a re-encapsulation script of the tcpdump tool for conveniently capturing the entire SMC communication process. smcdump-ex creates a virtual network device named smc-dummy{4-letter random character} in the current net namespace and enables the feature of capturing SMC data packets and control packets. smcdump-ex uses tcpdump to capture SMC handshake negotiation packets, data packets, and control packets on Ethernet devices or virtual network devices as required. Upon receiving a SIGINT (Ctrl+C) signal, smcdump-ex ends the capture, disables the capture feature, and destroys the virtual network device.

• Tool usage

  Run the following command to query the usage of smcdump-ex.

  Warning

  smcdump-ex is an experimental tool, and its usage may change in the future.

  smcdump-ex -h
  usage: smcdump-ex [-h] [-m {all,smc,smcd,smcr}] [-t {all,raw,cdc}]
                    [--param PARAM] [--filter FILTER] [--legacy]
  SMC Dump - SMC Traffic Capture (Experimental)
  optional arguments:
    -h, --help            show this help message and exit
    -m {all,smc,smcd,smcr}, --mode {all,smc,smcd,smcr}
                          Select the mode (default: smc)
    -t {all,raw,cdc}, --type {all,raw,cdc}
                          Select the packet type (default: all)
    --param PARAM         Additional parameters for tcpdump. e.g. --param '-w
                          packets.pcap'
    --filter FILTER       Additional filter expressions for tcpdump. e.g.
                          --filter 'host xxx.xxx.x.x and port 8080'
    --legacy              Use the legacy SMC dump header format

  • Use -m to set the data capture mode

    Optional capture modes:

    • all: Capture packets on all network interfaces, which is equivalent to invoking tcpdump -i any. If you specify all, SMC handshake negotiation packets on Ethernet interfaces and capture SMC data packets and control packets on newly created virtual network interfaces are captured. In this mode, we recommend that you use --filter to configure additional tcpdump filter expressions to narrow the scope of network packet capture for accurately capturing SMC negotiation packets. For more information, see --filter.

    • smc, smcr, and smcd: Capture only SMC, SMC-R, and SMC-D data packets and control packets on the newly created virtual network interfaces, respectively, which is equivalent to invoking tcpdump -i smc-dummy{4-letter random character}.

  • Use -t to set the type of network packets to capture

    Optional types of network packets to capture:

    • all: SMC data packets and control packets.

    • raw: only SMC data packets.

    • cdc: only SMC control packets.

  • Use --param to specify additional tcpdump parameters

    For example, specify --param '-w smcdata.pcap' to dump the captured content to a file.

  • Use --filter to configure additional tcpdump filter rules

    For example, specify --filter 'host <ip> and port <port>' to configure the IP address and port of the packets to be captured. This ensures that SMC negotiation packets are accurately captured and prevents interference from other TCP network packets in the result analysis.

  • Use --legacy to parse the old smc dump header format

    This parameter is only for compatibility with the ANCK 5.10.134-17.3 kernel version. For kernel versions ANCK 5.10.134-18 and later, this parameter is not required.

• Examples

  For example, run the following smcdump-ex command to capture all data packets that include the IP address 192.168.2.5 and port number 5201 during the SMC connection communication and dump the captured packets to the smc.pcap file:

  smcdump-ex -m all -t all --param '-w smc.pcap' --filter 'host 192.168.2.5 and port 5201'

  You can use Wireshark with the Lua plug-in to further analyze the captured packets that you dump to the .pcap file by using the --param '-w <pcap file>' parameter.

  1. Download and install Wireshark.

  2. Use Wireshark with the Lua plug-in to further analyze the captured packets stored in the .pcap file.

     1. Download the Lua plug-in: https://os-smc-new.oss-cn-hangzhou.aliyuncs.com/smc_dump.lua.

        Note

        To use the Lua plug-in, ensure that the Lua interpreter version in Wireshark is 5.3 or later.

     2. Check the installation path of the Wireshark Lua script and place the Lua script in the path.

        • Check the installation path of the Wireshark Lua script on macOS.

          1. Run Wireshark.

          2. In the menu bar, choose Wireshark > About Wireshark > Folders

             

             

        • Check the installation path of the Wireshark Lua script on Windows.

          1. Run Wireshark.

          2. In the main interface of Wireshark, choose Help > Folders.

             

  3. Restart Wireshark or reload the Lua plug-in for the script to take effect. Then, all UDP packets that include the SMC dump header in the payload are parsed as SMC packets.