Elastic Compute Service:Monitor and diagnose eRDMA

Possible results of a diagnose check

• PASS: The check passed.

• SKIP: The current system version does not support this check.

• FAIL: The check tool is not installed or the check failed. The failed command is listed in fail info.

• Other INFO messages: eRDMA configuration details, such as installation mode, driver version, and CC algorithm.

Install diagnose

On an eRDMA-configured instance, download the diagnose tool:

• Download from an internal URL

  wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

• Download from a public URL

  wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

[root@xxx                ~]# python diagnose.py -h
usage: diagnose.py [-h] [-d] [-D DIAGNOSES [DIAGNOSES ...]] [--debug]
                   [-H HOSTS [HOSTS ...]] [-p] [-P PERFTESTS [PERFTESTS ...]]
                   [-k KEY_FILE] [-s MESSAGE_SIZE] [-u USER] [-i ITERATIONS]
                   [--hpc-check] [--hpc-checks HPC_CHECKS [HPC_CHECKS ...]]

optional arguments:
  -h, --help            show this help message and exit
  -d, --diagnose        run all erdma basic diagnoses
  -D DIAGNOSES [DIAGNOSES ...], --diagnoses DIAGNOSES [DIAGNOSES ...]
                        list all basic diagnoses you want to run, separated by
                        space. Choices are: erdma device erdma installed erdma
                        loaded ibverbs loaded erdma tools hca detected hca
                        active erdma stats network config erdma dmesg atomic
                        support go-back-n support erdma install mode kernel
                        driver version rdma-core version cc algorithm
  --debug               print more detailed debug information
  -H HOSTS [HOSTS ...], --hosts HOSTS [HOSTS ...]
                        host list to run perftest, separated by space
  -p, --perftest        run all performance tests
  -P PERFTESTS [PERFTESTS ...], --perftests PERFTESTS [PERFTESTS ...]
                        list all perftests you want to run, separated by
                        space. Choices are: send_lat read_lat write_lat
  -k KEY_FILE, --key-file KEY_FILE
                        private key file path
  -s MESSAGE_SIZE, --message-size MESSAGE_SIZE
                        message size for perftest
  -u USER, --user USER  username for ssh connection
  -i ITERATIONS, --iterations ITERATIONS
                        iterations for perftest
  --hpc-check           check erdma related hpc configuration
  --hpc-checks HPC_CHECKS [HPC_CHECKS ...]
                        list all hpc diagnoses you want to run, separated by
                        space. Choices are: cc algorithm go-back-n support
                        delay-ack disable port dapl1 dapl2

Diagnose basic eRDMA functions

The basic functional test verifies driver installation, network connectivity, and the eRDMA kernel driver's installation mode.

Procedure:

1. Log on to the eRDMA-configured instance.

   See Connect to a Linux instance by using Workbench.

2. Download the diagnose tool.

   • Download from an internal URL

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • Download from a public URL

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. Check the basic functions of eRDMA:

   python diagnose.py -d

   The following is an example output. For diagnostic item descriptions, see eRDMA basic function diagnostic items.

   [root@jxxx            ~]# python diagnose.py -d
   [INFO] Begin to run erdma basic diagnoses. It's better to ensure that erdma driver and related tools, e.g. eadm, ibv_devinfo, rdma, have been installed.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result                          Desc                                                    fail info
   ------------------------------------------------------------------------------------------------------------------------------
        erdma device           PASS        erdma pci device is present
      erdma installed          PASS        erdma module has been installed
        erdma loaded           PASS        erdma module has been loaded
       ibverbs loaded          PASS        ib_uverbs module has been loaded
         erdma tools           PASS        eadm|rdma|ibv_devinfo tools have been installed
        hca detected           PASS        IB device exist
         hca active            PASS        status is PORT_ACTIVE
         erdma stats           PASS        no failed counter
       network config          PASS        network configuration seems ok
         erdma dmesg           PASS        no concerning error logs detected
       atomic support          PASS        atomic capacity support
     go-back-n support         FAIL        Go-back-N capacity support                    CHCK FAILED [eadm info -t ext_attr -d erdma_0 | grep -w attr_mask | awk '{print $2}']
     erdma install mode      Standard      erdma installation mode
   kernel driver version      0.2.37       erdma kernel driver version
     rdma-core version        44.1-2       rdma-core version
        cc algorithm          hpcc_rtt     current cc algorithm

Diagnose the eRDMA HPC environment

HPC applications in an eRDMA environment may require additional dependencies and configurations. The diagnose tool checks for these dependencies.

Procedure:

1. Log on to the eRDMA-configured instance.

   See Connect to a Linux instance by using Workbench.

2. Download the diagnose tool.

   • Download from an internal URL

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • Download from a public URL

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. Check HPC environment dependencies:

   python diagnose.py --hpc-check

   Example output. For diagnostic item descriptions, see eRDMA HPC environment dependency checks.

   ecs-user@xxx ~$ python diagnose.py --hpc-check
   [INFO] Begin to run hpc related diagnoses for erdma. If the missing dependencies are not required, please ignore them.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result              Desc                                          fail info
   ----------------------------  -------------------  -------------------------------------------------------  -------------------
       cc algorithm              cubic               current cc algorithm
     go-back-n support           PASS                Go-back-N capacity support
       dapl1 install             FAIL                dapl1 installation check                     CHCK FAILED [ls /etc/rdma/compat-dapl/dat.conf && ls /usr/lib64/libdaplcma.so.1.0.2]
       dapl2 install             PASS                dapl2 installation check
       dapl2 config              PASS                dapl2 erdma configuration check
       dapl2 test                PASS                dapl2 dtest result

Evaluate eRDMA network performance

Use the perftest feature in the diagnose tool to test network performance between instances.

• Prerequisites

  Before running the test:

  • eRDMA is installed and deployed on all test nodes. See Enable eRDMA for an ECS instance.

  • Passwordless SSH is configured among all test nodes. See Configure passwordless SSH logon.

  • Python paramiko is installed on all test nodes.

    Note

    • The diagnose tool uses paramiko for remote connections.

    • Install paramiko with the following commands. Python 3 is recommended.

    Alibaba Cloud Linux/CentOS

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko 
    # python2
    # If the pip module is not installed for Python 2, install python2-pip.
    sudo yum -y install python2-pip
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko 

    Ubuntu

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko
    # python2
    # If python2-pip is not installed on the current node, install it.
    sudo apt install software-properties-common
    sudo add-apt-repository universe
    sudo apt update
    sudo apt install python2
    sudo curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
    sudo python2 get-pip.py
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko

• Procedure

  1. Log on to the eRDMA-configured instance.

     See Connect to a Linux instance by using Workbench.

  2. Download the diagnose tool.

     • Download from an internal URL

       wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

     • Download from a public URL

       wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

  3. Test eRDMA latency:

     python diagnose.py --perftest --hosts <n1> <n2> --user <username> --key-file </path/to/private_key>

     Parameters:

     • --hosts <n1> <n2>: Test nodes, separated by spaces. Replace <n1> <n2> with the private IP address of the eRDMA-enabled ENI on each node.

     • --user <username>: Username for passwordless SSH.

     • --key-file </path/to/private_key>: Absolute path to the private key file for SSH.

     Example output for two-instance latency test. See eRDMA network performance tests.

     Each table shows latency for different operations. Rows represent requesters, columns represent responders. Cell values show average latency in microseconds (99.9th percentile in parentheses).

     [ecs-xxx                ~]$ python diagnose.py --perftest --hosts 172.17.0.87 172.17.0.86 --user ecs-user --key-file /home/ecs-user/.ssh/id_rsa
     /usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team.
     or Python 3.6.
       from cryptography.hazmat.backends import default_backend
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     [INFO] Begin to run perftest, the results will be shown in table.
     [INFO] In each table, the line represents client while the column represents server.
     [INFO] Each table cell shows the average latency and p999 latency(in the bracket).
     [INFO] Current perftest configuration:
     [INFO] tests: send_lat read_lat write_lat message_size: 2 iterations: 10000
     ----------------------------------------------------------------------
     |            ib_send_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   14.60(87.53)    |   14.29(25.25)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   14.35(51.63)    |    8.31(21.09)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_read_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   24.28(51.43)    |   27.57(57.42)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   27.43(41.14)    |   13.88(49.83)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_write_lat latency table(us)                          |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |  183.71(25003.59) |   12.76(27.00)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   12.80(26.05)    |    6.41(13.75)                 |
     ----------------------------------------------------------------------