This post delves into how we will simulate the real scenarios below:
The focus of this post is on Kafka and Apache RocketMQ™, since they both are high-performance applications. Thus, we will target these two products to see which one of them is better in the above scenarios. Before we begin, let’s clarify two concepts first:
Topic is an important concept in message-oriented middleware. Every topic represents a message category. With multiple topics, we can categorize and isolate messages.
You can refer to the feeding models in a zoo in the figure below. Every animal can only consume the specific corresponding food.
Both Kafka and Apache RocketMQ™ adopt message queues on disks. For the same consumption group, a partition only supports message consumption by one consumption thread. Too few partitions may cause the consumption speed to lag far behind the message generation speed. In actual production environments, a topic will be set in the multi-partition mode to support multiple consumers, as shown in the figure below:
Many topics and environments exist in an actual production environment of internet enterprises. Such production environment requires the message-oriented middleware to guarantee the service stability in the coexistence of multiple topics. Let's move on to the testing step to check the handling capacities of Kafka and Apache RocketMQ™ on multiple topics when message sending and subscription ends coexist.
We will compare the impact of the number of topics on the performances of Kafka and RocketMQ when sending and receiving ends coexist. In this case, we adopt eight partitions. However, in this stress test, we only pay attention to the performance indicators of the service end. Therefore, the “Exit” criteria for the stress test are as given below.
Keep increasing the stress on the sending end until the system throughput does not increase and the response takes longer. At this time, a performance bottleneck emerges on the service end. Obtain the corresponding optimal throughput of the system, ensuring that there are no messages accumulated throughout the process.
By default, every topic has eight partitions; every topic matches one subscriber, and the number of topics is increased gradually. The data obtained is as follows:
We can see that no matter how many topics there are, both Kafka and Apache RocketMQ™ can maintain flat TPS on the sending and receiving ends. Simply put, no message accumulation occurs.
Based on the changes in number of topics, we can draw the curves of message handling capacities of Kafka and Apache RocketMQ™, as shown in the figure below:
From the figure above we can see that:
Why the huge difference? The difference is attributable to the fact that every topic and partition of Kafka correspond to one physical file. When the number of topics increases, the policy of deconcentrated storage of messages to disks will lead to disk IO competition to cause performance bottlenecks. In contrast, all the messages in Apache RocketMQ™ are stored in the same physical file. The number of topics and partitions is just a logic division for Apache RocketMQ™. So the increasing number of topics won't generate a huge impact on the Apache RocketMQ™ performance.
When the message sending and consumption ends coexist, the increasing number of topics will cause a drastic decline of Kafka's throughput, while Apache RocketMQ™ delivers a stable performance. Therefore, Kafka is more suitable for business scenarios with only a few topics and consumption ends, while Apache RocketMQ™ is a better choice for business scenarios with multiple topics and consumption ends.
The service end is deployed in the standalone mode. The server configurations are as follows:
Application version:
In the test above, we saw how Apache RocketMQ™ almost overwhelms Kafka. The result is hardly surprising because Apache RocketMQ™ is born to target internet production requirements. Readers now should understand why Apache RocketMQ™ manages to support the massive messaging services of Alibaba group.
The stress test of multiple-topic scenarios involved in the test only lasted 20 minute. Too short an execution period for a message-oriented middleware product, hence inadequate to identify its stability.
2,605 posts | 747 followers
FollowAlibaba Cloud Native Community - January 9, 2024
Alibaba Developer - December 17, 2018
JJ Lim - January 11, 2022
Alibaba Cloud Native Community - February 1, 2023
Alibaba Cloud Native Community - December 6, 2022
Alibaba Clouder - August 5, 2020
2,605 posts | 747 followers
Follow
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn More
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn MoreMore Posts by Alibaba Clouder
Jantje Cloud May 8, 2018 at 4:04 am
Interesting!