All Products
Search

This Product

    Elastic Compute Service:Deploy high-performance bRPC applications with eRDMA

    Document Center

    Elastic Compute Service:Deploy high-performance bRPC applications with eRDMA

    Last Updated:Apr 27, 2026

    Deploy bRPC on eRDMA-capable ECS instances for lower latency, higher throughput, and reduced CPU usage.

    Note

    • bRPC is a high-performance, general-purpose RPC framework in C++. It provides tools to simplify service development and deployment. Common use cases include search, storage, machine learning, and advertising. bRPC suits high-concurrency, low-latency microservices and large-scale distributed systems. See the bRPC build instructions.

    • eRDMA (elastic Remote Direct Memory Access) is a cloud-native RDMA networking service that provides low latency, high throughput, and high elasticity. See Overview of eRDMA.

    Step 1: Prepare the environment

    Create two eRDMA-capable ECS instances as the server and client. Specify the following configuration items:

    • Instance type: Choose an eRDMA-capable instance type. See Limits. This topic uses ecs.g8a.8xlarge.

    • Image: Choose an eRDMA-compatible image. See Limits. This topic uses Alibaba Cloud Linux 3.2104 LTS 64-bit.

    • Install the eRDMA driver: Select Install eRDMA driver. The driver installs automatically during instance startup.

      Note

      After the instance starts, wait 3 to 5 minutes for the driver installation to complete. See Enable eRDMA on enterprise-level instances.

    • Networking:

      • Enable Internet access for both instances.

      • Service interconnection is enabled by default within the same VPC.

      • For ENI, select Elastic RDMA interface next to the NIC.

    For other parameters, see Customize and purchase an instance.

    Step 2: Deploy and compile bRPC

    Deploy and compile bRPC on both instances. The following steps use Alibaba Cloud Linux 3 as an example. For other operating systems, see Compile bRPC.

    1. Log on to both ECS instances. See Use Workbench to log on to a Linux instance.

    2. Modify the connection establishment mode for eRDMA and bRPC to ensure compatibility.

      Note

      By default, eRDMA uses RDMA_CM to establish connections, while bRPC uses Out-of-Band (OOB) mode. Modify the connection mode for both to ensure compatibility.

      sudo sh -c "echo 'options erdma compat_mode=Y' >> /etc/modprobe.d/erdma.conf"
sudo dracut --force
sudo rmmod erdma
sudo modprobe erdma compat_mode=Y

    3. Remove memory locking limits. eRDMA applications typically require large amounts of memory.

      1. Edit the limits.conf file.

        sudo vi /etc/security/limits.conf

      2. Add the following lines to the end of the file and save.

        * soft memlock unlimited
* hard memlock unlimited

    4. Deploy the bRPC application.

      sudo yum install git gcc-c++ make openssl-devel gflags-devel protobuf-devel protobuf-compiler leveldb-devel -y
git clone https://github.com/apache/brpc.git

    5. Apply the following patch on both instances to improve bRPC performance with eRDMA.

      1. In the brpc directory, create a file named erdma-multi-sge.patch:

        cd ~/brpc
sudo vi erdma-multi-sge.patch

      2. Add the following content and save.

        diff --git a/src/brpc/rdma/rdma_helper.cpp b/src/brpc/rdma/rdma_helper.cpp
index cf1cce95..d2592cbb 100644
--- a/src/brpc/rdma/rdma_helper.cpp
+++ b/src/brpc/rdma/rdma_helper.cpp
@@ -619,7 +619,7 @@ void DeregisterMemoryForRdma(void* buf) {
 }

 int GetRdmaMaxSge() {
-    return g_max_sge;
+    return 4;
 }

 int GetRdmaCompVector() {
--
2.39.3

      3. Apply the patch to the bRPC source code.

        patch -p1 < erdma-multi-sge.patch

      4. Compile the bRPC source code.

        sh config_brpc.sh --with-rdma --headers="/usr/include" --libs="/usr/lib64 /usr/bin"
make -j
cd example/rdma_performance; make -j

    Step 3: Test performance

    Test bRPC performance with and without eRDMA, then compare the results.

    1. Test bRPC performance in the following two scenarios.

      With eRDMA enabled

      1. On the server, start the server with eRDMA enabled.

        ./server --rdma_gid_index=1 --rdma_prepared_qp_cnt=0 --use_rdma=true

      2. On the client, connect to the server with eRDMA enabled.

        ./client --servers=<server private IP address>:8002 --rpc_timeout_ms=-1 --attachment_size=1024 --rdma_gid_index=1 --rdma_prepared_qp_cnt=0 --use_rdma=true --queue_depth=16

        Parameters:

        • Replace <server private IP address> with the server's private IP address.

        • --attachment_size: the data size attached to each bRPC call. Larger values better leverage eRDMA but increase memory management complexity. Set based on your test requirements.

        • --queue_depth: the number of concurrent requests the queue can hold. A higher depth helps handle traffic spikes but consumes more memory. Set based on your test requirements.

      Without eRDMA enabled

      1. On the server, start the server with TCP.

        ./server --rdma_gid_index=1 --rdma_prepared_qp_cnt=0 --use_rdma=false

      2. On the client, connect to the server with TCP.

        ./client --servers=<server private IP address>:8002 --rpc_timeout_ms=-1 --attachment_size=1024 --rdma_gid_index=1 --rdma_prepared_qp_cnt=0 --use_rdma=false --queue_depth=16

        Parameters:

        • Replace <server private IP address> with the server's private IP address.

        • --attachment_size: the data size attached to each bRPC call. Larger values better leverage eRDMA but increase memory management complexity. Set based on your test requirements.

        • --queue_depth: the number of concurrent requests the queue can hold. A higher depth helps handle traffic spikes but consumes more memory. Set based on your test requirements.

    2. Compare the results from both scenarios. Check the Avg-Latency field for latency and the QPS field for throughput to evaluate eRDMA performance gains.