Deploy an Elastic High Performance Computing cluster with eRDMA - Elastic High Performance Computing

Deploy an E-HPC cluster with elastic Remote Direct Memory Access (eRDMA) support, verify the RDMA environment, and run OSU-Benchmark tests to measure latency and bandwidth gains over TCP.

Background

Multi-node parallel HPC workloads — such as climate forecasting, industrial simulation, and molecular dynamics — depend on high bandwidth and low latency to run efficiently. eRDMA delivers on-premises-level network performance in the cloud without additional network interface controllers (NICs), so the cluster integrates into your existing network seamlessly.

Prerequisites

Before you begin, make sure you have:

An Alibaba Cloud account with E-HPC access
Permissions to create ECS instances and E-HPC clusters in the Shanghai region

Step 1: Set up the cluster

Create an E-HPC cluster

Go to the Create Cluster page and create a cluster using the following configuration. For full setup instructions, see Create a Standard Edition cluster.

Section	Setting	Value
Cluster configuration	Region	Shanghai
Network and zone	Zone	Zone L
	Series	Standard Edition
	Deployment mode	Public Cloud Cluster
	Cluster type	SLURM
Control plane node	Instance type	ecs.c7.xlarge (4 vCPUs, 8 GiB memory)
	Image	aliyun_2_1903_x64_20G_alibase_20240628.vhd
Compute nodes and queues	Number of queue nodes	2 (initial)
	Inter-node interconnection	eRDMA Network
	Instance type	ecs.c8ae.xlarge or other AMD instances of the same generation
	Image	aliyun_2_1903_x64_20G_alibase_20240628.vhd
Shared file storage	Cluster mount directories	`/home` and `/opt` (mounted to a shared file system by default)
Software and service components	Software to install	erdma-installer, mpich-aocc
Installable service components	Logon node instance type	ecs.c7.xlarge (4 vCPUs, 8 GiB memory)
	Logon node image	aliyun_2_1903_x64_20G_alibase_20240628.vhd

Note

Only some node specifications support Elastic RDMA Interconnect (ERI). For details, see elastic Remote Direct Memory Access (eRDMA) and Enable eRDMA on enterprise-level instances.

Note

The OSU-Benchmark package in this tutorial is built on the Alibaba Cloud Linux 2.1903 LTS 64-bit image. Use this image for both control plane and compute nodes.

Create a cluster user

Create a cluster user to run jobs. For instructions, see User management.

Step 2: Verify the eRDMA environment

Before running benchmark tests, confirm that eRDMA is configured correctly on all compute nodes.

Log on to the Elastic High Performance Computing console and click the cluster you created.
Go to Nodes & Queues > Nodes, select all compute nodes, and click Send Command.
Send the following command to all compute nodes to check the eRDMA network status and hardware/software support:
```
hpcacc erdma check
```
If the check returns a passing result, the eRDMA configuration is correct. Proceed to Step 3. If an error is returned, run the following command to repair the configuration:
```
hpcacc erdma repair
```
After the repair completes, run hpcacc erdma check again to confirm the issue is resolved. If the problem persists, contact technical support and provide the full output of hpcacc erdma check.

Step 3: Run OSU-Benchmark tests

OSU-Benchmark measures point-to-point communication performance across different network protocols. This tutorial uses two benchmarks:

osu_latency — measures one-way latency for small messages (1 B to a few KB). Low latency directly reduces communication overhead in real-time simulations and parameter synchronization for distributed ML jobs.
osu_bw — measures sustainable bandwidth for larger messages (a few KB to several MB). Bandwidth performance affects large data transfers such as matrix exchanges in scientific computing.

Both benchmarks run with eRDMA (Verbs protocol) and TCP so you can compare the two directly.

Run the benchmark

Connect to the cluster as the user you created.
Confirm the required environment modules are installed:
```
module avail
```

Download and extract the precompiled OSU-Benchmark package:

cd ~ && wget https://ehpc-perf.oss-cn-hangzhou.aliyuncs.com/AMD-Genoa/osu-bin.tar.gz
tar -zxvf osu-bin.tar.gz

Navigate to the point-to-point test directory and open the Slurm job script:
```
cd ~/pt2pt
vim slurm.job
```

Replace the contents of slurm.job with the following script:

Parameter	Description	Value used
`-np`	Total number of MPI processes	`2`
`-ppn`	Processes per node	`1` (one process per compute node)
`-genv`	Environment variable applied to all processes	—
`FI_PROVIDER="verbs;ofi_rxm"`	Verbs protocol (RDMA-based) with the RXM reliability layer. Use this for eRDMA performance.	eRDMA tests
`FI_PROVIDER="tcp;ofi_rxm"`	TCP with the RXM reliability layer. Use this as the baseline for comparison.	TCP tests

#!/bin/bash
#SBATCH --job-name=osu-bench
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=2
#SBATCH --partition=comp
#SBATCH --output=%j.out
#SBATCH --error=%j.out

# Load environment parameters
module purge
module load aocc/4.0.0 gcc/12.3.0 libfabric/1.16.0 mpich-aocc/4.0.3

# Run MPI latency test: eRDMA
echo -e "++++++ use erdma for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_latency
echo -e "------ use erdma for osu_lat: END\n"
# Run MPI latency test: TCP
echo -e "++++++ use tcp for osu_lat: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_latency
echo -e "------ use tcp for osu_lat: END\n"

# Run MPI bandwidth test: eRDMA
echo -e "++++++ use erdma for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="verbs;ofi_rxm" ./osu_bw
echo -e "------ use erdma for osu_bw: END\n"
# Run MPI bandwidth test: TCP
echo -e "++++++ use tcp for osu_bw: START"
mpirun -np 2 -ppn 1 -genv FI_PROVIDER="tcp;ofi_rxm" ./osu_bw
echo -e "------ use tcp for osu_bw: END\n"

The script runs all four tests sequentially. The key mpirun parameters are: For eRDMA performance, use FI_PROVIDER="verbs;ofi_rxm". For more information, see Alibaba Cloud eRDMA.

Submit the job:
```
sbatch slurm.job
```
The output shows the job ID.
Check the job status:
```
squeue
```
You can also monitor storage, job, and node status in the E-HPC console. For details, see View monitoring information.

Review the results

When the job completes, open the log file named {job-id}.out in the ~/pt2pt directory.

Latency results (osu_latency)

The output shows one-way latency in microseconds for message sizes from 1 B to 4 MB.

eRDMA (Verbs protocol)

TCP

For small messages (1 B to 8 KB), eRDMA latency is significantly lower than TCP.

Bandwidth results (osu_bw)

The output shows bandwidth in MB/s for message sizes from 1 B to 4 MB.

eRDMA (Verbs protocol)

TCP

For message sizes from 16 KB to 64 KB, eRDMA fully utilizes available network bandwidth. TCP introduces additional processing overhead at the same message sizes.