Elastic Compute Service:Enable eRDMA on an enterprise-level instance

Test whether the eRDMA network latency meets expectations.

1. On the server instance, run the following command to start ib_write_lat as a server and listen for client connections.

   ib_write_lat -R -a -F

   • -R: Specifies that the RDMA Connection Manager (RDMA_CM) is used to establish a connection.

     Important

     • For CPU-based instance types that support eRDMA, the default installation mode for the eRDMA kernel driver is Standard, which supports only the RDMA_CM connection establishment method. For more information, see Connection establishment methods.

     • By default, perftest uses an out-of-band (OOB) connection. When performing a perftest on a CPU-based instance, you can use the -R parameter on both the server and the client to specify the RDMA_CM connection establishment method. Otherwise, an exception may occur when the connection is established.

     • You can also use the command line to make the RDMA_CM and OOB connection methods compatible. For more information, see Change the connection establishment mode of eRDMA to be compatible with bRPC. After you enable compatibility, the -R parameter is no longer required.

   • -a: This option runs tests for all message sizes from 2 to 2^23 bytes. This allows you to test the impact of different message sizes on latency.

   • -F: Forcefully overrides any existing connections. This means that if connections were previously established, using the -F option ignores these connections and forces new connections to be established.

2. On the client instance, run the following command to start ib_write_lat and connect to the server.

   ib_write_lat -R -a -F <server_ip>

   <server_ip> is the private IP address of the network interface on the server ECS instance that is used for the test and has the ERI feature enabled. To obtain the IP address, see View IP addresses.

3. View the test results.

   After the client test is complete, ib_write_lat outputs test configuration information, connection information, and performance test results (statistics related to latency, including performance metrics such as minimum, maximum, and average latency).

   

   Description of the ib_write_lat test result fields

   • #bytes: The size of the message payload used in the test. The size ranges from 2 bytes to 8,388,608 bytes. Different message sizes can help you understand performance under different loads.

   • #iterations: The number of iterations. This value specifies how many times the test for each message size is repeated. A higher number of iterations can provide more stable averages and statistics.

   • t_min[usec]: The minimum latency, in microseconds, recorded across all measurements. This value provides a reference for the best-case network latency.

   • t_max[usec]: The maximum latency in microseconds, representing the highest latency recorded across all measurements. A high maximum latency may indicate certain issues in the network or transient congestion.

   • t_typical[usec]: The typical latency, in microseconds. This value represents the latency commonly observed in a test and is usually the median of all measured values.

   • t_avg[usec]: The average latency of all measurements, in microseconds. This value gives you an overall impression of the network latency.

   • t_stdev[usec]: The standard deviation of latency, in microseconds. This value indicates the degree of variation in latency. A smaller standard deviation means that the latency is more stable, while a larger standard deviation indicates greater fluctuations.

   • 99% percentile[usec]: The 99th percentile latency, in microseconds. This means that 99% of the measured values are lower than this value. These data points can help you understand latency performance in extreme cases.

   • 99.9% percentile[usec]: The 99.9th percentile latency, in microseconds. This indicates that 99.9% of the measured values are lower than this value. These data points help you understand latency performance in extreme cases.

   Analyzing these metrics helps you understand your RDMA network's performance for optimization and troubleshooting. For example, if you notice a sudden increase in latency at a specific message size, you may need to check whether your network configuration or hardware performance meets the requirements. If you see significant latency fluctuations, you may need to investigate network congestion or other instability factors.