Deploy an eRDMA-enabled E-HPC cluster - Elastic High Performance Computing

Background information

Using eRDMA, multi-node parallel High-Performance Computing (HPC) workloads on an E-HPC cluster (formerly E-HPC NEXT), such as climate and weather modeling, industrial simulation, and molecular dynamics, can achieve network performance comparable to on-premises clusters. This includes high bandwidth and low latency, which significantly improves the efficiency of numerical simulations. You can use RDMA within your existing network without deploying additional RDMA NICs. This approach provides seamless integration and is easy to use.

Prerequisites

To create an E-HPC cluster, go to the Create Cluster page. For more information, see Create a Standard Edition cluster.

The following table describes the sample cluster configuration.

	Parameter	Configuration
Cluster Configuration	Region	China (Shanghai)
	Network and Availability Zone	Select Availability Zone L
	Series	Standard Edition
	Deployment Mode	Public cloud cluster
	Cluster Type	SLURM
	Management node	Instance Specification: ecs.c7.xlarge with 4 vCPUs and 8 GiB of memory. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd Note The osu-benchmark installation package is built based on the Alibaba Cloud Linux 2.1903 LTS 64-bit image.
Compute Node and Queue	Queue Compute Nodes	Initial nodes: `2`.
	Inter-node interconnection	eRDMA network Note Only specific instance types support Elastic RDMA Interconnect (ERI). For more information, see Elastic RDMA (eRDMA) and Enable eRDMA on enterprise-level instances.
	Instance type Group	Instance Specification: ecs.c8ae.xlarge or other AMD instances of the same generation. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd
Shared File Storage	/home cluster mount directory	By default, a file system is mounted to the `/home` and `/opt` directories on the management node to serve as shared storage directories.
Shared File Storage	/opt mount directory
Software and Service Component	Software not Installed	erdma-installer mpich-aocc
Software and Service Component	Installable service components	Logon Node: Instance Specification: ecs.c7.xlarge with 4 vCPUs and 8 GiB of memory. Image: aliyun_2_1903_x64_20G_alibase_20240628.vhd

Create a cluster user. For more information, see user management.

Check the eRDMA environment

Verify the eRDMA configuration on the compute nodes.

Log on to the Elastic High Performance Computing console and click the target cluster.
In the left-side navigation pane, choose Nodes and Queues > Node. Select all compute nodes in the cluster and click Send Command.
Check the eRDMA network status and RDMA support on the compute nodes.
1. Send the following command to all compute nodes.
```
hpcacc erdma check
```
  In the Cloud Assistant Send Command dialog box, select the target instance. Set Execution Plan to Immediate, Command Type to Shell, and Timeout to 60 seconds. Click Run.
2. A result similar to the following indicates that the eRDMA configuration is correct.
```
{"Success": true, "Message": {"Msg": "HPCACC eRDMA status check succeeded. No fault found."}}
```
3. If an error is returned, run the following command to repair the configuration.
```
hpcacc erdma repair
```
4. After the repair is complete, verify the eRDMA configuration again.

Run OSU-Benchmark tests

OSU-Benchmark evaluates the communication performance of High-Performance Computing (HPC) clusters and distributed systems. This topic uses the following two benchmarks to test communication performance over different network protocols (TCP vs. RDMA):

Latency test (osu_latency): Measures the one-way latency for point-to-point communication (the time from when process A sends a message to when process B receives it, excluding the return time). The test focuses on small messages from 1 byte to several kilobytes (KB). Small-message latency, a core metric for HPC system responsiveness, reflects low-level network performance (such as RDMA acceleration capabilities) and MPI library optimization. For example, in real-time simulations or machine learning parameter synchronization, low latency can significantly reduce communication overhead.
Bandwidth test (osu_bw): Measures the sustainable point-to-point bandwidth (the amount of data transferred per unit time). The test focuses on large messages from several kilobytes (KB) to several megabytes (MB). Bandwidth performance directly affects the efficiency of large data transfers, such as matrix exchange in scientific computing or file I/O scenarios. If the measured bandwidth is far below the theoretical value, you may need to optimize MPI configurations (such as multithreaded communication) or check network settings (such as MTU and flow control).

Procedure

As the user you created, connect to the E-HPC cluster. For more information, see Connect to a cluster.
Run the following command to check whether the dependent environment modules are correctly installed.
```
module avail
```

Run the following commands to download and decompress the pre-compiled OSU-Benchmark installation package.

cd ~ && wget https://ehpc-perf.oss-cn-hangzhou.aliyuncs.com/AMD-Genoa/osu-bin.tar.gz
tar -zxvf osu-bin.tar.gz

Run the following commands to go to the test directory and edit the Slurm job script.

cd ~/pt2pt 
vim slurm.job

The content of the test script is as follows:

#!/bin/bash
#SBATCH --job-name=osu-bench
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=2
#SBATCH --partition=comp
#SBATCH --output=%j.out
#SBATCH --error=%j.out

# load env params
module purge
module load aocc/4.0.0 gcc/12.3.0 libfabric/1.16.0 mpich-aocc/4.0.3

# run mpi latency test: erdma
echo -e "++++++ use erdma for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_latency
echo -e "------ use erdma for osu_lat: END\n"
# run mpi latency test: tcp
echo -e "++++++ use tcp for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_latency
echo -e "------ use erdma for osu_lat: END\n"

# run mpi bandwidth test: erdma
echo -e "++++++ use erdma for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_bw
echo -e "------ use erdma for osu_bw: END\n"
# run mpi bandwidth test: tcp
echo -e "++++++ use tcp for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_bw
echo -e "------ use tcp for osu_bw: END\n"

Note

-np 2: Specifies the total number of processes. In this case, it is set to 2, which means the MPI job starts two processes.
-ppn 1: Specifies the number of processes per node. In this case, it is set to 1, which means one process runs on each node.
-genv: Sets environment variables that apply to all processes.
- FI_PROVIDER="tcp;ofi_rxm": Uses the TCP protocol and enhances communication reliability through the RXM framework.
- FI_PROVIDER="verbs;ofi_rxm": Prioritizes the high-performance Verbs protocol, which is based on RDMA, and optimizes message transmission through the RXM framework. In this topic, Alibaba Cloud eRDMA provides a high-performance elastic RDMA network.

Run the following command to submit the test job.
```
sbatch slurm.job
```
The command-line interface returns the job ID.

Run the following command to view the job status. During the test, you can also view monitoring information for the E-HPC cluster, such as storage, job, and node metrics, in the console. For more information, see View monitoring information.

squeue

[zh***luster@login0 pt2pt]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                11      comp slurm.jo zhtestcl  R       0:01      2 compute[000-001]

In the current directory, you can view the log file that corresponds to the job ID. The output is similar to the following content:

Network latency test results: The results show the relationship between the message size (Size in bytes, from 1 B to 4 MB) and the average network latency (Avg Latency).

Verbs protocol (eRDMA)

[zxxx@login0 pt2pt]$ cat 11.out
++++++ use erdma for osu_lat: START

# OSU MPI Latency Test v7.5
# Datatype: MPI_CHAR.
# Size          Avg Latency(us)
1                       11.01
2                       11.04
4                       11.48
8                       11.33
16                      11.02
32                      11.10
64                      11.15
128                     11.29
256                     11.36
512                     11.39
1024                    11.81
2048                    12.99
4096                    13.95
8192                    15.92
16384                   34.57
32768                   37.15
65536                   41.15
131072                  47.10
262144                  61.17
524288                 143.06
1048576                332.54
2097152                659.65
4194304               1315.29
------ use erdma for osu_lat: END

TCP protocol

++++++ use tcp for osu_lat: START

# OSU MPI Latency Test v7.5
# Datatype: MPI_CHAR.
# Size          Avg Latency(us)
1                          16.07
2                          16.02
4                          16.03
8                          16.03
16                         16.01
32                         16.27
64                         16.37
128                        16.87
256                        17.36
512                        17.89
1024                       18.56
2048                       33.78
4096                       33.39
8192                       30.74
16384                      45.17
32768                      94.50
65536                     107.07
131072                    124.95
262144                    160.55
524288                    238.40
1048576                   374.89
2097152                   639.81
4194304                  1188.98
------ use erdma for osu_lat: END

The test data indicates that for small messages (1 B to 8 KB), eRDMA latency is significantly lower than TCP latency.

Network bandwidth test results: The results show the relationship between the message size (Size in bytes, from 1 B to 4 MB) and the bandwidth (Bandwidth).

Verbs protocol (eRDMA)

++++++ use erdma for osu_bw: START

# OSU MPI Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size          Bandwidth (MB/s)
1                       1.09
2                       2.21
4                       4.43
8                       8.91
16                     17.98
32                     35.88
64                     69.93
128                   137.73
256                   258.35
512                   541.90
1024                 1023.92
2048                 1768.96
4096                 2918.97
8192                 3984.86
16384                5203.51
32768                6264.41
65536                1661.85
131072               1696.79
262144               1535.17
524288               1587.83
1048576              1637.65
2097152              1636.63
4194304              1637.17

------ use erdma for osu_bw: END

TCP protocol

++++++ use tcp for osu_bw: START

# OSU MPI Bandwidth Test v7.5
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.38
2                       0.75
4                       1.51
8                       3.01
16                      5.95
32                     12.10
64                     23.43
128                    46.30
256                    88.25
512                   167.70
1024                  343.96
2048                  614.60
4096                 1183.85
8192                 1567.03
16384                1835.89
32768                1731.67
65536                2904.03
131072               3160.96
262144               1876.96
524288               1730.89
1048576              1733.18
2097152              1736.09
4194304              1735.78
------ use tcp for osu_bw: END

The test data indicates that for message sizes from 16 KB to 64 KB, eRDMA fully utilizes the network bandwidth, whereas the TCP protocol stack introduces additional overhead.