云服务器 ECS：监测和诊断eRDMA

diagnose检测可能存在以下几种结果

• PASS：通过当前检测。

• SKIP：当前系统版本不支持该检测，跳过该检测。

• FAIL：相应检测工具未安装或检测失败，检测命令会在fail info中列出。

• 其他INFO信息：eRDMA相关配置信息，如安装模式、驱动版本以及CC算法等。

安装diagnose

在已配置eRDMA的实例上，执行以下命令，获取diagnose工具。

• 从内网地址获取

  wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

• 从公网地址获取

  wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

[root@xxx                ~]# python diagnose.py -h
usage: diagnose.py [-h] [-d] [-D DIAGNOSES [DIAGNOSES ...]] [--debug]
                   [-H HOSTS [HOSTS ...]] [-p] [-P PERFTESTS [PERFTESTS ...]]
                   [-k KEY_FILE] [-s MESSAGE_SIZE] [-u USER] [-i ITERATIONS]
                   [--hpc-check] [--hpc-checks HPC_CHECKS [HPC_CHECKS ...]]

optional arguments:
  -h, --help            show this help message and exit
  -d, --diagnose        run all erdma basic diagnoses
  -D DIAGNOSES [DIAGNOSES ...], --diagnoses DIAGNOSES [DIAGNOSES ...]
                        list all basic diagnoses you want to run, separated by
                        space. Choices are: erdma device erdma installed erdma
                        loaded ibverbs loaded erdma tools hca detected hca
                        active erdma stats network config erdma dmesg atomic
                        support go-back-n support erdma install mode kernel
                        driver version rdma-core version cc algorithm
  --debug               print more detailed debug information
  -H HOSTS [HOSTS ...], --hosts HOSTS [HOSTS ...]
                        host list to run perftest, separated by space
  -p, --perftest        run all performance tests
  -P PERFTESTS [PERFTESTS ...], --perftests PERFTESTS [PERFTESTS ...]
                        list all perftests you want to run, separated by
                        space. Choices are: send_lat read_lat write_lat
  -k KEY_FILE, --key-file KEY_FILE
                        private key file path
  -s MESSAGE_SIZE, --message-size MESSAGE_SIZE
                        message size for perftest
  -u USER, --user USER  username for ssh connection
  -i ITERATIONS, --iterations ITERATIONS
                        iterations for perftest
  --hpc-check           check erdma related hpc configuration
  --hpc-checks HPC_CHECKS [HPC_CHECKS ...]
                        list all hpc diagnoses you want to run, separated by
                        space. Choices are: cc algorithm go-back-n support
                        delay-ack disable port dapl1 dapl2

eRDMA基础功能诊断

您可以通过diagnose进行eRDMA的基础功能诊断测试，包括驱动是否正确安装、网络连通性是否正常、eRDMA内核驱动的安装模式等，以确保 eRDMA 功能可以正常运行并发挥其高吞吐量和低延迟的优势。

具体操作如下所示：

1. 远程登录已配置eRDMA的实例。

   具体操作，请参见使用Workbench登录Linux实例。

2. 执行以下命令，获取diagnose工具。

   • 从内网地址获取

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • 从公网地址获取

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. 执行以下命令，检测eRDMA基础功能。

   python diagnose.py -d

   结果如下所示，可以看到本示例中测试项结果。关于诊断项说明，请参见eRDMA基础功能诊断项说明。

   [root@jxxx            ~]# python diagnose.py -d
   [INFO] Begin to run erdma basic diagnoses. It's better to ensure that erdma driver and related tools, e.g. eadm, ibv_devinfo, rdma, have been installed.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result                          Desc                                                    fail info
   ------------------------------------------------------------------------------------------------------------------------------
        erdma device           PASS        erdma pci device is present
      erdma installed          PASS        erdma module has been installed
        erdma loaded           PASS        erdma module has been loaded
       ibverbs loaded          PASS        ib_uverbs module has been loaded
         erdma tools           PASS        eadm|rdma|ibv_devinfo tools have been installed
        hca detected           PASS        IB device exist
         hca active            PASS        status is PORT_ACTIVE
         erdma stats           PASS        no failed counter
       network config          PASS        network configuration seems ok
         erdma dmesg           PASS        no concerning error logs detected
       atomic support          PASS        atomic capacity support
     go-back-n support         FAIL        Go-back-N capacity support                    CHCK FAILED [eadm info -t ext_attr -d erdma_0 | grep -w attr_mask | awk '{print $2}']
     erdma install mode      Standard      erdma installation mode
   kernel driver version      0.2.37       erdma kernel driver version
     rdma-core version        44.1-2       rdma-core version
        cc algorithm          hpcc_rtt     current cc algorithm

eRDMA HPC环境诊断

如果您需要在eRDMA的环境下运行HPC应用，您可能需要安装一些额外的依赖和配置，diagnose工具提供eRDMA HPC环境依赖项检测功能。

具体操作如下所示：

1. 远程登录已配置eRDMA的实例。

   具体操作，请参见使用Workbench登录Linux实例。

2. 执行以下命令，获取diagnose工具。

   • 从内网地址获取

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • 从公网地址获取

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. 执行如下命令，检测eRDMA HPC环境依赖项。

   python diagnose.py --hpc-check

   正常情况下，执行上述检测命令的输出结果如下。关于诊断项说明，请参见eRDMA HPC环境检测依赖项说明。

   ecs-user@xxx ~$ python diagnose.py --hpc-check
   [INFO] Begin to run hpc related diagnoses for erdma. If the missing dependencies are not required, please ignore them.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result              Desc                                          fail info
   ----------------------------  -------------------  -------------------------------------------------------  -------------------
       cc algorithm              cubic               current cc algorithm
     go-back-n support           PASS                Go-back-N capacity support
       dapl1 install             FAIL                dapl1 installation check                     CHCK FAILED [ls /etc/rdma/compat-dapl/dat.conf && ls /usr/lib64/libdaplcma.so.1.0.2]
       dapl2 install             PASS                dapl2 installation check
       dapl2 config              PASS                dapl2 erdma configuration check
       dapl2 test                PASS                dapl2 dtest result

eRDMA网络性能评估

您可以使用diagnose的perftest功能进行不同实例之间的网络性能测试。

• 前提条件

  在执行检测之前，您需要确保：

  • 已在所有待检测节点上正确安装和部署eRDMA。具体操作，请参见ECS实例启用eRDMA。

  • 已配置所有待检测节点之间可通过SSH免密连接。具体操作，请参见配置SSH免密登录。

  • 已在所有待检测节点上安装Python paramiko依赖包。

    说明

    • diagnose利用paramiko实现远程连接。

    • 您可以参考如下命令安装Python paramiko依赖包。如果您对Python版本没有要求，推荐您默认使用Python3以减少配置工作量。

    Alibaba Cloud Linux/Centos

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko 
    # python2
    # 如果当前python2未安装pip模块，请安装python2-pip
    sudo yum -y install python2-pip
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko 

    Ubuntu

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko
    # python2
    # 如果当前节点未安装python2-pip
    sudo apt install software-properties-common
    sudo add-apt-repository universe
    sudo apt update
    sudo apt install python2
    sudo curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
    sudo python2 get-pip.py
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko

• 示例操作

  1. 远程登录已配置eRDMA的实例。

     具体操作，请参见使用Workbench登录Linux实例。

  2. 执行以下命令，获取diagnose工具。

     • 从内网地址获取

       wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

     • 从公网地址获取

       wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

  3. 执行如下命令，进行eRDMA延迟检测。

     python diagnose.py --perftest --hosts <n1> <n2> --user <username> --key-file </path/to/private_key>

     各参数说明如下：

     • --hosts <n1> <n2>：待检测的各节点，彼此之间通过空格分隔。您需要将<n1> <n2>替换为检测节点上的eRDMA弹性网卡对应的私有IP地址。

     • --user <username>：SSH免密登录用户名，请根据实际环境替换。

     • --key-file </path/to/private_key>：SSH免密登录所使用的私钥文件路径（绝对路径），请根据实际环境替换。

     检测结果示例如下，显示了两台实例之间的网络延迟测试。详细信息，请参见eRDMA网络性能测试。

     每个表格展示了不同操作下的延迟情况。表格中的行代表发起请求的一方，列代表响应请求的一方。每个单元格内的数值表示平均延迟（单位为微秒），括号内是第99.9百分位的延迟时间。

     [ecs-xxx                ~]$ python diagnose.py --perftest --hosts 172.17.0.87 172.17.0.86 --user ecs-user --key-file /home/ecs-user/.ssh/id_rsa
     /usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team.
     or Python 3.6.
       from cryptography.hazmat.backends import default_backend
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     [INFO] Begin to run perftest, the results will be shown in table.
     [INFO] In each table, the line represents client while the column represents server.
     [INFO] Each table cell shows the average latency and p999 latency(in the bracket).
     [INFO] Current perftest configuration:
     [INFO] tests: send_lat read_lat write_lat message_size: 2 iterations: 10000
     ----------------------------------------------------------------------
     |            ib_send_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   14.60(87.53)    |   14.29(25.25)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   14.35(51.63)    |    8.31(21.09)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_read_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   24.28(51.43)    |   27.57(57.42)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   27.43(41.14)    |   13.88(49.83)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_write_lat latency table(us)                          |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |  183.71(25003.59) |   12.76(27.00)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   12.80(26.05)    |    6.41(13.75)                 |
     ----------------------------------------------------------------------