Elastic Compute Service：監測和診斷eRDMA

diagnose檢測可能存在以下幾種結果

• PASS：通過當前檢測。

• SKIP：當前系統版本不支援該檢測，跳過該檢測。

• FAIL：相應偵查工具未安裝或檢測失敗，檢測命令會在fail info中列出。

• 其他INFO資訊：eRDMA相關配置資訊，如安裝模式、驅動版本以及CC演算法等。

安裝diagnose

在已配置eRDMA的執行個體上，執行以下命令，擷取diagnose工具。

• 從內網地址擷取

  wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

• 從公網地址擷取

  wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

[root@xxx                ~]# python diagnose.py -h
usage: diagnose.py [-h] [-d] [-D DIAGNOSES [DIAGNOSES ...]] [--debug]
                   [-H HOSTS [HOSTS ...]] [-p] [-P PERFTESTS [PERFTESTS ...]]
                   [-k KEY_FILE] [-s MESSAGE_SIZE] [-u USER] [-i ITERATIONS]
                   [--hpc-check] [--hpc-checks HPC_CHECKS [HPC_CHECKS ...]]

optional arguments:
  -h, --help            show this help message and exit
  -d, --diagnose        run all erdma basic diagnoses
  -D DIAGNOSES [DIAGNOSES ...], --diagnoses DIAGNOSES [DIAGNOSES ...]
                        list all basic diagnoses you want to run, separated by
                        space. Choices are: erdma device erdma installed erdma
                        loaded ibverbs loaded erdma tools hca detected hca
                        active erdma stats network config erdma dmesg atomic
                        support go-back-n support erdma install mode kernel
                        driver version rdma-core version cc algorithm
  --debug               print more detailed debug information
  -H HOSTS [HOSTS ...], --hosts HOSTS [HOSTS ...]
                        host list to run perftest, separated by space
  -p, --perftest        run all performance tests
  -P PERFTESTS [PERFTESTS ...], --perftests PERFTESTS [PERFTESTS ...]
                        list all perftests you want to run, separated by
                        space. Choices are: send_lat read_lat write_lat
  -k KEY_FILE, --key-file KEY_FILE
                        private key file path
  -s MESSAGE_SIZE, --message-size MESSAGE_SIZE
                        message size for perftest
  -u USER, --user USER  username for ssh connection
  -i ITERATIONS, --iterations ITERATIONS
                        iterations for perftest
  --hpc-check           check erdma related hpc configuration
  --hpc-checks HPC_CHECKS [HPC_CHECKS ...]
                        list all hpc diagnoses you want to run, separated by
                        space. Choices are: cc algorithm go-back-n support
                        delay-ack disable port dapl1 dapl2

eRDMA基礎功能診斷

您可以通過diagnose進行eRDMA的基礎功能診斷測試，包括驅動是否正確安裝、網路連通性是否正常、eRDMA核心驅動的安裝模式等，以確保 eRDMA 功能可以正常運行並發揮其高輸送量和低延遲的優勢。

具體操作如下所示：

1. 遠程登入已配置eRDMA的執行個體。

   具體操作，請參見使用Workbench登入Linux執行個體。

2. 執行以下命令，擷取diagnose工具。

   • 從內網地址擷取

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • 從公網地址擷取

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. 執行以下命令，檢測eRDMA基礎功能。

   python diagnose.py -d

   結果如下所示，可以看到本樣本中測試項結果。關於診斷項說明，請參見eRDMA基礎功能診斷項說明。

   [root@jxxx            ~]# python diagnose.py -d
   [INFO] Begin to run erdma basic diagnoses. It's better to ensure that erdma driver and related tools, e.g. eadm, ibv_devinfo, rdma, have been installed.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result                          Desc                                                    fail info
   ------------------------------------------------------------------------------------------------------------------------------
        erdma device           PASS        erdma pci device is present
      erdma installed          PASS        erdma module has been installed
        erdma loaded           PASS        erdma module has been loaded
       ibverbs loaded          PASS        ib_uverbs module has been loaded
         erdma tools           PASS        eadm|rdma|ibv_devinfo tools have been installed
        hca detected           PASS        IB device exist
         hca active            PASS        status is PORT_ACTIVE
         erdma stats           PASS        no failed counter
       network config          PASS        network configuration seems ok
         erdma dmesg           PASS        no concerning error logs detected
       atomic support          PASS        atomic capacity support
     go-back-n support         FAIL        Go-back-N capacity support                    CHCK FAILED [eadm info -t ext_attr -d erdma_0 | grep -w attr_mask | awk '{print $2}']
     erdma install mode      Standard      erdma installation mode
   kernel driver version      0.2.37       erdma kernel driver version
     rdma-core version        44.1-2       rdma-core version
        cc algorithm          hpcc_rtt     current cc algorithm

eRDMA HPC環境診斷

如果您需要在eRDMA的環境下運行HPC應用，您可能需要安裝一些額外的依賴和配置，diagnose工具提供eRDMA HPC環境依賴項檢測功能。

具體操作如下所示：

1. 遠程登入已配置eRDMA的執行個體。

   具體操作，請參見使用Workbench登入Linux執行個體。

2. 執行以下命令，擷取diagnose工具。

   • 從內網地址擷取

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • 從公網地址擷取

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. 執行如下命令，檢測eRDMA HPC環境依賴項。

   python diagnose.py --hpc-check

   正常情況下，執行上述檢測命令的輸出結果如下。關於診斷項說明，請參見eRDMA HPC環境檢測依賴項說明。

   ecs-user@xxx ~$ python diagnose.py --hpc-check
   [INFO] Begin to run hpc related diagnoses for erdma. If the missing dependencies are not required, please ignore them.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result              Desc                                          fail info
   ----------------------------  -------------------  -------------------------------------------------------  -------------------
       cc algorithm              cubic               current cc algorithm
     go-back-n support           PASS                Go-back-N capacity support
       dapl1 install             FAIL                dapl1 installation check                     CHCK FAILED [ls /etc/rdma/compat-dapl/dat.conf && ls /usr/lib64/libdaplcma.so.1.0.2]
       dapl2 install             PASS                dapl2 installation check
       dapl2 config              PASS                dapl2 erdma configuration check
       dapl2 test                PASS                dapl2 dtest result

eRDMA網路效能評估

您可以使用diagnose的perftest功能進行不同執行個體之間的網路效能測試。

• 前提條件

  在執行檢測之前，您需要確保：

  • 已在所有待檢測節點上正確安裝和部署eRDMA。具體操作，請參見ECS執行個體啟用eRDMA。

  • 已配置所有待檢測節點之間可通過SSH免密串連。具體操作，請參見配置SSH免密登入。

  • 已在所有待檢測節點上安裝Python paramiko依賴包。

    說明

    • diagnose利用paramiko實現遠端連線。

    • 您可以參考如下命令安裝Python paramiko依賴包。如果您對Python版本沒有要求，推薦您預設使用Python3以減少配置工作量。

    Alibaba Cloud Linux/Centos

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko 
    # python2
    # 如果當前python2未安裝pip模組，請安裝python2-pip
    sudo yum -y install python2-pip
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko 

    Ubuntu

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko
    # python2
    # 如果當前節點未安裝python2-pip
    sudo apt install software-properties-common
    sudo add-apt-repository universe
    sudo apt update
    sudo apt install python2
    sudo curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
    sudo python2 get-pip.py
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko

• 樣本操作

  1. 遠程登入已配置eRDMA的執行個體。

     具體操作，請參見使用Workbench登入Linux執行個體。

  2. 執行以下命令，擷取diagnose工具。

     • 從內網地址擷取

       wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

     • 從公網地址擷取

       wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

  3. 執行如下命令，進行eRDMA延遲檢測。

     python diagnose.py --perftest --hosts <n1> <n2> --user <username> --key-file </path/to/private_key>

     各參數說明如下：

     • --hosts <n1> <n2>：待檢測的各節點，彼此之間通過空格分隔。您需要將<n1> <n2>替換為檢測節點上的eRDMA彈性網卡對應的私人IP地址。

     • --user <username>：SSH免密登入使用者名稱，請根據實際環境替換。

     • --key-file </path/to/private_key>：SSH免密登入所使用的私密金鑰檔案路徑（絕對路徑），請根據實際環境替換。

     檢測結果樣本如下，顯示了兩台執行個體之間的網路延遲測試。詳細資料，請參見eRDMA網路效能測試。

     每個表格展示了不同操作下的延遲情況。表格中的行代表發起請求的一方，列代表響應請求的一方。每個儲存格內的數值表示平均延遲（單位為微秒），括弧內是第99.9百分位的延遲時間。

     [ecs-xxx                ~]$ python diagnose.py --perftest --hosts 172.17.0.87 172.17.0.86 --user ecs-user --key-file /home/ecs-user/.ssh/id_rsa
     /usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team.
     or Python 3.6.
       from cryptography.hazmat.backends import default_backend
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     [INFO] Begin to run perftest, the results will be shown in table.
     [INFO] In each table, the line represents client while the column represents server.
     [INFO] Each table cell shows the average latency and p999 latency(in the bracket).
     [INFO] Current perftest configuration:
     [INFO] tests: send_lat read_lat write_lat message_size: 2 iterations: 10000
     ----------------------------------------------------------------------
     |            ib_send_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   14.60(87.53)    |   14.29(25.25)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   14.35(51.63)    |    8.31(21.09)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_read_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   24.28(51.43)    |   27.57(57.42)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   27.43(41.14)    |   13.88(49.83)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_write_lat latency table(us)                          |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |  183.71(25003.59) |   12.76(27.00)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   12.80(26.05)    |    6.41(13.75)                 |
     ----------------------------------------------------------------------