Elastic Compute Service:Deploy a Kafka cluster with eRDMA for high-throughput, low-latency messaging

Step 1: Prepare ECS instances

Prepare ECS instances for Broker, ZooKeeper, and stress testing before deploying the Kafka cluster.

• A Broker instance stores, transmits, and manages messages as a core data node.

• A ZooKeeper instance handles distributed coordination for the Kafka cluster.

• A stress testing instance benchmarks the Kafka cluster performance.

This example uses five ECS instances: one ZooKeeper, three Broker, and one stress testing instance. The following table lists the configuration requirements.

Step 2: Install required tools and Kafka

Log on to each ECS instance from Step 1 and install SMC-R, Java, and Kafka.

1. Log on to all ECS instances.

   See Connect to a Linux instance by using a password or key.

2. (Conditionally required) Run uname -r to check the kernel version. Ensure the kernel version is 5.10.134-16.3 or later. If earlier than 5.10.134-16.3, upgrade the kernel:

   sudo yum update kernel
   sudo reboot

3. Install smc-tools on each instance:

   sudo yum install smc-tools -y

4. Verify that eRDMA is enabled on each instance:

   smcr dev

   Sample output:

   Net-Dev IB-Dev IB-P IB-State Type Crit #Links PNET-ID 
   eth0 erdma_0 1 ACTIVE 0x107f No 0 

5. Disable IPv6 on each instance.

   Note

   Alibaba Cloud eRDMA and SMC devices do not support IPv6. Disabling IPv6 ensures traffic flows through IPv4 RDMA channels.

   sudo sysctl net.ipv6.conf.all.disable_ipv6=1

6. Install Java and Git:

   sudo yum install java-11-openjdk-1:11.0.21.0.9-2.0.3.al8 java-11-openjdk-devel-1:11.0.21.0.9-2.0.3.al8 git -y

7. Download and decompress the Kafka package:

   wget https://archive.apache.org/dist/kafka/3.5.0/kafka_2.13-3.5.0.tgz
   tar -xf kafka_2.13-3.5.0.tgz

Step 3: Start ZooKeeper and Broker for Kafka

1. Log on to all ECS instances.

2. Add the mapping between the private IP address and hostname of each instance to the /etc/hosts file.

   Sample configuration:

   ::1         localhost       localhost.localdomain    localhost6      localhost6.localdomain6
   127.0.0.1   localhost       localhost.localdomain    localhost4      localhost4.localdomain4
   
   72.17.xxx.xxx iZ2zead6q4xxx iZ2zead6q4xxx
   72.17.xxx.xxx iZ2zeeaqc9xxx iZ2zeeaqc9xxx
   72.17.xxx.xxx iZ2zeeaqc9 iZ2zeeaqc9xxx
   72.17.xxx.xxx iZ2zeeaqc9 iZ2zeeaqc9xxx
   72.17.xxx.xxx iZ2ze2s4vkxxx iZ2ze2s4vkxxx

3. Start ZooKeeper on the ZooKeeper instance:

   bash $HOME/kafka_2.13-3.5.0/bin/zookeeper-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/zookeeper.properties

4. Start Broker on each Broker instance.

   Note

   To test without eRDMA, remove the smc_run parameter from the command.

   • On the first Broker instance, set the Broker ID to 0 and start Broker. Replace <zookeeper ip> with the private IP address of the ZooKeeper instance.

     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=0 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181

   • On the second Broker instance, set the Broker ID to 1 and start Broker. Replace <zookeeper ip> with the private IP address of the ZooKeeper instance.

     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=1 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181

   • On the third Broker instance, set the Broker ID to 2 and start Broker. Replace <zookeeper ip> with the private IP address of the ZooKeeper instance.

     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=2 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181

Step 4: Test Kafka performance

Download the Benchmark tool and configure it for maximum network bandwidth. Test Kafka performance with and without eRDMA and compare the results.

1. Log on to the stress testing instance, then download and compile Open Messaging Benchmark.

   1. Download and install Maven, the compiler for Open Messaging Benchmark.

      wget https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
      tar -xf apache-maven-3.8.8-bin.tar.gz
      export PATH=$PATH:$HOME/apache-maven-3.8.8/bin/

   2. Configure the Maven mirror to accelerate downloads.

      vi $HOME/apache-maven-3.8.8/conf/settings.xml

      Add the following content to the settings.xml mirrors tag, then save and close the file.

      <mirror>
          <id>nexus-aliyun</id>
          <mirrorOf>central</mirrorOf>
          <name>Nexus aliyun</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public</url>
      </mirror>

   3. Download and compile Open Messaging Benchmark:

      git clone https://github.com/openmessaging/benchmark.git
      cd benchmark && mvn clean verify -DskipTests

2. Configure the Broker IP addresses in the kafka-throughput.yaml file.

   vi $HOME/benchmark/driver-kafka/kafka-throughput.yaml

   Set bootstrap.servers to the private IP addresses of the Broker instances: <Private IP address of Broker 0>:9092,<Private IP address of Broker 1>:9092,<Private IP address of Broker 2>:9092.

   commonConfig: |
     bootstrap.servers=<172.17.XX.XX>:9092,<172.17.XX.XX>:9092,<172.17.XX.XX>:9092
     default.api.timeout.ms=1200000
     request.timeout.ms=1200000

3. Set the message sending rate to saturate the available network bandwidth.

   vi $HOME/benchmark/workloads/1-topic-100-partitions-1kb-4p-4c-200k.yaml

   Change producerRate: <Message sending rate> . The rate is calculated as: Available bandwidth of a Broker-enabled instance/Size of a single message.

   In this example, each Broker uses ecs.g8a.2xlarge (max 4 Gbit/s). The total bandwidth of three Brokers is 12 Gbit/s. Due to Kafka's triplicate mechanism, the available bandwidth of a Broker-enabled instance is 12/3 = 4 Gbit/s (512 MB/s). With a 1 KB message size in the workloads directory, the message sending rate is 512 MB/s / 1 KB = 524,288. Set producerRate: <Message sending rate> to producerRate: 524288. Adjust the rate based on your actual requirements.

4. Test Kafka cluster performance using one of the following methods.

   Perform a performance test when eRDMA is enabled

   smc_run $HOME/benchmark/bin/benchmark --drivers $HOME/benchmark/driver-kafka/kafka-throughput.yaml $HOME/benchmark/workloads/1-topic-100-partitions-1kb-4p-4c-2000k.yaml

   During the test, you can also:

   • Run smcss -a in another window on the stress testing instance to verify SMC-R is transmitting messages.

   • Run sar on each Broker instance to check CPU utilization. For example, sar 1 20 samples once per second for 20 times. Sum the CPU utilization of all three Broker instances for the total.

   Perform a performance test when eRDMA is disabled

   $HOME/benchmark/bin/benchmark --drivers $HOME/benchmark/driver-kafka/kafka-throughput.yaml $HOME/benchmark/workloads/1-topic-100-partitions-1kb-4p-4c-2000k.yaml

   Important

   To prevent residual test data from affecting the non-eRDMA test, delete Broker and ZooKeeper records and restart both services.

   Delete and restart Broker and ZooKeeper

   • Stop ZooKeeper and delete its records:

     bash $HOME/kafka_2.13-3.5.0/bin/zookeeper-server-stop.sh
     rm -rf /tmp/zookeeper/

   • Restart ZooKeeper:

     bash $HOME/kafka_2.13-3.5.0/bin/zookeeper-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/zookeeper.properties

   • Stop Broker on each Broker instance and delete the records:

     bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-stop.sh
     rm -rf $HOME/kafka-logs/

   • Restart Broker on each Broker instance:

     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=0 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181
     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=1 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181
     KAFKA_HEAP_OPTS="-Xmx4G -Xms4G" smc_run bash $HOME/kafka_2.13-3.5.0/bin/kafka-server-start.sh -daemon $HOME/kafka_2.13-3.5.0/config/server.properties --override broker.id=2 --override log.dirs=$HOME/kafka-logs --override zookeeper.connect=<zookeeper ip>:2181

     • Replace <zookeeper ip> with the private IP address of the ZooKeeper instance.

     • To test without eRDMA, remove the smc_run parameter from the command.

5. Compare the latency results from both tests to evaluate eRDMA's performance impact.

   Find the last Aggregated Pub Latency (ms) entry. avg is the average latency, 99% is the P99 latency, and 999% is the P999 latency.