Elastic Compute Service:Pantau dan diagnosa eRDMA

Kemungkinan hasil pemeriksaan diagnose

• PASS: Pemeriksaan berhasil.

• SKIP: Versi sistem saat ini tidak mendukung pemeriksaan ini.

• FAIL: Tool pemeriksaan tidak terinstal atau pemeriksaan gagal. Perintah yang gagal tercantum dalam fail info.

• Pesan INFO lainnya: Detail konfigurasi eRDMA, seperti mode instalasi, versi driver, dan algoritma CC.

Instal diagnose

Pada instans yang telah dikonfigurasi eRDMA, unduh tool diagnose:

• Unduh dari URL internal

  wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

• Unduh dari URL publik

  wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

[root@xxx                ~]# python diagnose.py -h
usage: diagnose.py [-h] [-d] [-D DIAGNOSES [DIAGNOSES ...]] [--debug]
                   [-H HOSTS [HOSTS ...]] [-p] [-P PERFTESTS [PERFTESTS ...]]
                   [-k KEY_FILE] [-s MESSAGE_SIZE] [-u USER] [-i ITERATIONS]
                   [--hpc-check] [--hpc-checks HPC_CHECKS [HPC_CHECKS ...]]

optional arguments:
  -h, --help            show this help message and exit
  -d, --diagnose        run all erdma basic diagnoses
  -D DIAGNOSES [DIAGNOSES ...], --diagnoses DIAGNOSES [DIAGNOSES ...]
                        list all basic diagnoses you want to run, separated by
                        space. Choices are: erdma device erdma installed erdma
                        loaded ibverbs loaded erdma tools hca detected hca
                        active erdma stats network config erdma dmesg atomic
                        support go-back-n support erdma install mode kernel
                        driver version rdma-core version cc algorithm
  --debug               print more detailed debug information
  -H HOSTS [HOSTS ...], --hosts HOSTS [HOSTS ...]
                        host list to run perftest, separated by space
  -p, --perftest        run all performance tests
  -P PERFTESTS [PERFTESTS ...], --perftests PERFTESTS [PERFTESTS ...]
                        list all perftests you want to run, separated by
                        space. Choices are: send_lat read_lat write_lat
  -k KEY_FILE, --key-file KEY_FILE
                        private key file path
  -s MESSAGE_SIZE, --message-size MESSAGE_SIZE
                        message size for perftest
  -u USER, --user USER  username for ssh connection
  -i ITERATIONS, --iterations ITERATIONS
                        iterations for perftest
  --hpc-check           check erdma related hpc configuration
  --hpc-checks HPC_CHECKS [HPC_CHECKS ...]
                        list all hpc diagnoses you want to run, separated by
                        space. Choices are: cc algorithm go-back-n support
                        delay-ack disable port dapl1 dapl2

Diagnosa fungsi dasar eRDMA

Pengujian fungsi dasar memverifikasi instalasi driver, konektivitas jaringan, dan mode instalasi driver kernel eRDMA.

Prosedur:

1. Masuk ke instans yang telah dikonfigurasi eRDMA.

   Lihat Hubungkan ke instans Linux menggunakan Workbench.

2. Unduh tool diagnose.

   • Unduh dari URL internal

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • Unduh dari URL publik

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. Periksa fungsi dasar eRDMA:

   python diagnose.py -d

   Berikut adalah contoh output. Untuk deskripsi item diagnostik, lihat Item diagnostik fungsi dasar eRDMA.

   [root@jxxx            ~]# python diagnose.py -d
   [INFO] Begin to run erdma basic diagnoses. It's better to ensure that erdma driver and related tools, e.g. eadm, ibv_devinfo, rdma, have been installed.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result                          Desc                                                    fail info
   ------------------------------------------------------------------------------------------------------------------------------
        erdma device           PASS        erdma pci device is present
      erdma installed          PASS        erdma module has been installed
        erdma loaded           PASS        erdma module has been loaded
       ibverbs loaded          PASS        ib_uverbs module has been loaded
         erdma tools           PASS        eadm|rdma|ibv_devinfo tools have been installed
        hca detected           PASS        IB device exist
         hca active            PASS        status is PORT_ACTIVE
         erdma stats           PASS        no failed counter
       network config          PASS        network configuration seems ok
         erdma dmesg           PASS        no concerning error logs detected
       atomic support          PASS        atomic capacity support
     go-back-n support         FAIL        Go-back-N capacity support                    CHCK FAILED [eadm info -t ext_attr -d erdma_0 | grep -w attr_mask | awk '{print $2}']
     erdma install mode      Standard      erdma installation mode
   kernel driver version      0.2.37       erdma kernel driver version
     rdma-core version        44.1-2       rdma-core version
        cc algorithm          hpcc_rtt     current cc algorithm

Diagnosa lingkungan HPC eRDMA

Aplikasi HPC dalam lingkungan eRDMA mungkin memerlukan dependensi dan konfigurasi tambahan. Tool diagnose memeriksa dependensi tersebut.

Prosedur:

1. Masuk ke instans yang telah dikonfigurasi eRDMA.

   Lihat Hubungkan ke instans Linux menggunakan Workbench.

2. Unduh tool diagnose.

   • Unduh dari URL internal

     wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

   • Unduh dari URL publik

     wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

3. Periksa dependensi lingkungan HPC:

   python diagnose.py --hpc-check

   Contoh output. Untuk deskripsi item diagnostik, lihat Pemeriksaan dependensi lingkungan HPC eRDMA.

   ecs-user@xxx ~$ python diagnose.py --hpc-check
   [INFO] Begin to run hpc related diagnoses for erdma. If the missing dependencies are not required, please ignore them.
   [INFO] The diagnose results mean:
   [INFO] PASS: pass the test or meet the requirement.
   [INFO] SKIP: encounter some problems when running the diagnose.
   [INFO] FAIL: test failed or requirements are not met.
   [INFO] XXXX: the erdma-related information, e.g. erdma driver version, cc algorithm.
           Name                Result              Desc                                          fail info
   ----------------------------  -------------------  -------------------------------------------------------  -------------------
       cc algorithm              cubic               current cc algorithm
     go-back-n support           PASS                Go-back-N capacity support
       dapl1 install             FAIL                dapl1 installation check                     CHCK FAILED [ls /etc/rdma/compat-dapl/dat.conf && ls /usr/lib64/libdaplcma.so.1.0.2]
       dapl2 install             PASS                dapl2 installation check
       dapl2 config              PASS                dapl2 erdma configuration check
       dapl2 test                PASS                dapl2 dtest result

Evaluasi kinerja jaringan eRDMA

Gunakan fitur perftest dalam tool diagnose untuk menguji kinerja jaringan antar instans.

• Prasyarat

  Sebelum menjalankan pengujian:

  • eRDMA telah diinstal dan diterapkan pada semua node pengujian. Lihat Aktifkan eRDMA untuk instans ECS.

  • SSH tanpa kata sandi telah dikonfigurasi di antara semua node pengujian. Lihat Konfigurasikan login SSH tanpa kata sandi.

  • Python paramiko telah diinstal pada semua node pengujian.

    Catatan

    • Tool diagnose menggunakan paramiko untuk koneksi jarak jauh.

    • Instal paramiko dengan perintah berikut. Disarankan menggunakan Python 3.

    Alibaba Cloud Linux/CentOS

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko 
    # python2
    # Jika modul pip belum diinstal untuk Python 2, instal python2-pip.
    sudo yum -y install python2-pip
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko 

    Ubuntu

    # python3
    sudo python3 -m pip install --upgrade pip
    sudo python3 -m pip install paramiko
    # python2
    # Jika python2-pip belum diinstal pada node saat ini, instal terlebih dahulu.
    sudo apt install software-properties-common
    sudo add-apt-repository universe
    sudo apt update
    sudo apt install python2
    sudo curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
    sudo python2 get-pip.py
    sudo python2 -m pip install --upgrade pip==20.3.4
    sudo python2 -m pip install paramiko

• Prosedur

  1. Masuk ke instans yang telah dikonfigurasi eRDMA.

     Lihat Hubungkan ke instans Linux menggunakan Workbench.

  2. Unduh tool diagnose.

     • Unduh dari URL internal

       wget http://mirrors.cloud.aliyuncs.com/erdma/tools/diagnose.py

     • Unduh dari URL publik

       wget https://mirrors.aliyun.com/erdma/tools/diagnose.py

  3. Uji latensi eRDMA:

     python diagnose.py --perftest --hosts <n1> <n2> --user <username> --key-file </path/to/private_key>

     Parameter:

     • --hosts <n1> <n2>: Node pengujian, dipisahkan dengan spasi. Ganti <n1> <n2> dengan alamat IP pribadi ENI yang diaktifkan eRDMA pada masing-masing node.

     • --user <username>: Username untuk SSH tanpa kata sandi.

     • --key-file </path/to/private_key>: Path absolut ke file kunci privat untuk SSH.

     Contoh output pengujian latensi dua instans. Lihat Pengujian kinerja jaringan eRDMA.

     Setiap tabel menunjukkan latensi untuk operasi berbeda. Baris merepresentasikan pengirim permintaan, kolom merepresentasikan penerima respons. Nilai sel menunjukkan latensi rata-rata dalam mikrodetik (persentil 99,9 dalam tanda kurung).

     [ecs-xxx                ~]$ python diagnose.py --perftest --hosts 172.17.0.87 172.17.0.86 --user ecs-user --key-file /home/ecs-user/.ssh/id_rsa
     /usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team.
     or Python 3.6.
       from cryptography.hazmat.backends import default_backend
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.0)
     INFO:paramiko.transport:Authentication (publickey) successful!
     [INFO] Begin to run perftest, the results will be shown in table.
     [INFO] In each table, the line represents client while the column represents server.
     [INFO] Each table cell shows the average latency and p999 latency(in the bracket).
     [INFO] Current perftest configuration:
     [INFO] tests: send_lat read_lat write_lat message_size: 2 iterations: 10000
     ----------------------------------------------------------------------
     |            ib_send_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   14.60(87.53)    |   14.29(25.25)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   14.35(51.63)    |    8.31(21.09)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_read_lat latency table(us)                           |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |   24.28(51.43)    |   27.57(57.42)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   27.43(41.14)    |   13.88(49.83)                 |
     ----------------------------------------------------------------------
     
     ----------------------------------------------------------------------
     |            ib_write_lat latency table(us)                          |
     ----------------------------------------------------------------------
     |               |    172.17.0.87    |    172.17.0.86                 |
     ----------------------------------------------------------------------
     |  172.17.0.87  |  183.71(25003.59) |   12.76(27.00)                 |
     ----------------------------------------------------------------------
     |  172.17.0.86  |   12.80(26.05)    |    6.41(13.75)                 |
     ----------------------------------------------------------------------