By Kaiyi & Yuntian
Proofreading: Baiyu
Major features are becoming available with the release of the RocketMQ 5.0 preview. POP Consumer, a key feature of 5.0, presents a new consumer model. It is lightweight, stateless, and no queue-exclusive. It is friendly to message backlog and Streaming consumption scenarios. Before introducing POP Consumer, let's review the most commonly used Push consumers.
People familiar with RocketMQ will be familiar with Push Consumer. Client-side consumption generally uses this consumption mode, and it is relatively simple to use. All we need to do is set up and write the business logic in the callback method ConsumeMessage and start the client application to consume messages.
public class PushConsumer {
public static void main(String[] args) throws InterruptedException, MQClientException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1");
consumer.subscribe("test_topic", "*");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}
}
How does Push Consumer consume messages?
The Consumer only receives the message if the Producer sends it to the Topic first. The Producer uses polling to send messages to each Queue separately. Generally, there is more than one consumer. When the client starts, it will be SLB in the Topic and Consumer group dimension and assign a Queue to be processed to each client. During the SLB process, each client can get all ConsumerID and queues and sort them. Each client uses the same Load Balancing Algorithm, such as the uniform allocating algorithm, so each client calculates which Queues it needs to consume. SLB will be triggered whenever the consumer increases or decreases. Therefore, we can implement dynamic expansion through the RocketMQ SLB mechanism to improve the client's ability to send and receive messages.
Can the number of clients be increased all the time to improve the consumption capacity? Of course not, because the number of Queues is limited. Once the number of clients reaches the number of Queues, expanding new nodes fails to improve the consumption capacity because some nodes cannot be allocated to Queues.
After client-side load balancing assigns Queue to the client, the client will keep pulling messages to the broker and consume at the client. Isn't it a Push client? How can it be that the client pulls messages to the broker? Shouldn't it be that the broker pushes messages to the client? This is an interesting point. In RocketMQ, all messages are consumed by client pulling whether it is Push Consumer, Pull Consumer, or POP Consumer (to be introduced later). Push Consumer only makes us feel pushed by Broker through encapsulation at the client API level.
After the client-side load balancing and pulling messages, the client can consume messages normally.
The complete processing logic of Push Consumer can be seen in the figure above.
First, the client Rebalance determines which consumer clients process which Queues and then pulls messages through the PullMessageService service. After pulling the messages, ConsumeMessageConcurrentlyService submits a consumption request to the message consumption thread pool. Then, call the callback method ConsumeMessage, where the message processing service can be obtained. Finally, the consumption updates the local offset and reports the offset to the Broker. If the consumption fails (exception throwing, timeout, etc.), the client will send sendBack to tell the broker which messages failed to be consumed. The broker sends the failed messages to the delayed queue and then puts them into the retry topic after the delay. The client consumes the retry topic to complete the retry. The advantage is that the consumption of normal messages will not be affected by failed messages. If you want to know the details, you can go to GitHub to download the source code and compare this figure to see the code processing flow.
The introduction of Push Consumer gives us a certain understanding of the principle of Push Consumer. RocketMQ clients do many things, such as SLB, message pulling, consumer offset management, and sendBack after failed consumption. This is unfriendly to multilingual support. People that have participated in multilingual development will feel the same way. It is difficult to transplant so much logic to different languages. This increases the difficulty of upgrading and O&M on the client side.
Do we think about whether we can slim down the client and move part of the logic from the client to the Broker? Of course, it is possible. From the previous introduction, we know when the Push Consumer client is responsible for balancing, the information SLB needs and all ConsumerId were originally obtained by the client from the Broker. All Queue information can be obtained by Broker through nameServer. There is no big difference between the client and the broker side to call the algorithm, so porting Rebalance to the Broker is a good choice. The Broker SLB can achieve the same effect as the client SLB. The client logic will be reduced, the multi-language implementation will be simpler, and the subsequent upgrade operation and maintenance will be more controllable. In addition, since the Broker has global information compared with the client. It can do some more interesting things, such as when responsible for balancing, executing SLB according to the backlog of Queue and assigning Queue from stressful clients to other clients for processing.
We learned some features of Push Consumer through the introduction of Push Consumer.
The preceding features may cause some problems. For example, an abnormal machine hangs on the client, which leads to the accumulation of allocation queue messages, thus failing to consume.
Push consumer consumption in RocketMQ is unfriendly when the machine is abnormal. If the client machine hangs but connects to the Broker, the client Rebalance will allocate a consumption queue to the hanging machine. When the consumption speed of the hanging machine is slow or cannot consume, there will be consumption accumulation. There are also unavoidable problems, such as consumption delay caused by Rebalance of the client when the server Broker is released. It is shown in the following figure:
When the Push Consumer 2 machine hangs, Q2 on the Broker is piled up. We deal with this problem by finding this machine to restart it or make it offline. We need to ensure that the business is not affected by abnormal machines. However, if the queue is squeezed to some extent, the machine may be unable to quickly catch up with the consumption progress after recovering, which is also limited by the ability of Push Consumer.
Let's summarize some pain points of Push Consumer:
Based on the preceding issues, RocketMQ 5.0 implements a new consumption model-POP Consumer.
POP Consumer can solve the preceding stability problem and relieve the extended capability of queue occupancy.
Let's take a brief look at how POP Consumer consumes messages:
The POP Client issues a POP request message from the Broker's queue, and the Broker returns messages. In the massage system attribute, there is an important one called POP_CK, which is the handler of a message. A message can be located through one handler. After the message consumption is successful, the POP client sends ackMessages and passes the handler to the broker to confirm the message consumption is successful.
For message retry, when POP sends out a message, this message will enter an invisible time, during which time it will not be POP again. If the message consumption is not confirmed through ackMessage during this invisible time, it will be visible again after the invisible time has passed.
In addition, the policy is a gradient delay time, and the retry gap gradually increases for the retry of the message. There is another changeInvisibleTime that can modify the invisible time of the message.
As shown in the figure above, the message would have been visible again in the middle of the time, but we used the changeInvisibleTime in advance to extend the invisible time and delay the visible time of this message. When the user service code returns reconsumeLater or throws exceptions, we can modify the next visible time based on the number of retries through changeInvisibleTime. In addition, if the consumption RT exceeds 30 seconds (the default value which can be modified), the broker will also put the message into the retry queue.
Moreover, POP consumption is stored and controlled by brokers. POP consumption can be consumed by multiple clients in the same queue, as shown in the following figure:
The three clients do not need Rebalance to allocate Queues. Instead, they all use POP to request all brokers to obtain messages for consumption. Even if POP Consumer 2 hangs, its internal messages will be consumed by POP Consumer 1 and POP Consumer 3. Then, the problem of consumption accumulation caused by hanging machines is solved.
As seen in the overall process, POP consumption can avoid the consumption delay caused by Rebalance, and the client can consume all the queues of the broker. This way, the problem of accumulation caused by machine hanging can be avoided.
At the same time, the expansion capability is improved. POP Consumer can consume all queues under the same topic. Compared with Push Consumer, the limit that each Queue must be rebalanced to one client consumption is lifted. The number of Push Consumer clients must be less than or equal to the number of Queues. POP consumers can break this limit. Multiple POP consumers can consume the same queue.
How is POP Consumer implemented on the broker side?
After a POP Consumer pulls a message, it locks it in the Queue dimension to ensure that only one client can pull messages to the same Queue at the same time. After the message is obtained, the checkPoint information will be stored in the broker. The checkPoint information mainly includes messages, such as Topic, ConsumerGroup, QueueId, offset, POPTime, msgCout, and reviveQueueId. The checkPoint information is preferentially saved to the buffer, waiting for the ack message and receiving the ack message replied by the client within a period. The corresponding checkPoint information is removed from the buffer, and the consumption progress is updated to identify that the message consumption is successful.
When the checkPoint message waits in the buffer and does not get the ack message, it will be cleared out of the buffer and will send a ck msg to the store. The ck msg will be sent to the delay queue SCHEDULE_Topic_XXXX first and will enter the REVIVE_LOG Topic after the delay is completed. The REVIVE_LOG Topic is the topic of ck msg and ack msg stored to be processed. POPReceiveService pulls the REVIVE_LOG Topic message into a map. If ck has a corresponding ack, the consumer offset of REVIVE_LOG will be updated to identify the ck msg whose message consumption has been completed but not been confirmed after a timeout. The real message corresponding to the ck msg will be queried, and this message will be put into the retry topic, waiting for the client to consume. POP Consumer will probably consume the message in the retry topic when it is normally consumed. You can see from this design that the common design of RocketMQ is used to implement business logic, transactional messages, and scheduled messages through some internal topics.
Let's briefly summarize the benefits of POP Consumer:
Since POP has so many advantages, can we use POP to solve some problems of Push? As we mentioned earlier, when a queue has accumulated a lot of consumer problems, Push Consumer is limited by the consumption-ability of a single consumer and cannot quickly catch up with the consumption progress. The delay will be high. The core problem is the limitation of single-queue and single-consumer, which makes the consumption capacity unable to scale out.
We hope that when a queue is piled up, we can switch to POP mode through the POPAPI form and have the opportunity for multiple consumers to consume the queue together and catch up with the progress. We also achieved this in the 5.0 version.
You can switch in two ways:
1. Command Line
mqadmin setConsumeMode -c cluster -t topic -g group -m POP -n 8
2. Code Switching
public static final String CONSUMER_GROUP = "CID_JODIE_1";
public static final String TOPIC = "TopicTest";
// Or use AdminTools directly: mqadmin setConsumeMode -c cluster -t topic -g group -m POP -n 8
private static void switchPop() throws Exception {
DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
mqAdminExt.start();
ClusterInfo clusterInfo = mqAdminExt.examineBrokerClusterInfo();
Set<String> brokerAddrs = clusterInfo.getBrokerAddrTable().values().stream().map(BrokerData::selectBrokerAddr).collect(Collectors.toSet());
for (String brokerAddr : brokerAddrs) {
mqAdminExt.setMessageRequestMode(brokerAddr, TOPIC, CONSUMER_GROUP, MessageRequestMode.POP, 8, 3_000);
}
}
We can see that POP Consumer and Push API are unified and simple to use through the following POP Consumer Demo. Compared with Push API, it is one more step of consumption mode switching.
When using POP consumption mode, we only need to switch modes based on Push API. We still need to do some processing for the Broker. Retry queue is the main thing that needs to be handled.
Push and POP modes process retry queues differently:
After the mode is switched, the messages in the retry of the old mode need to be processed. Otherwise, the messages will be lost.
In summary, we will handle the retry queue without mode switching.
POP is a new consumption mode that solves some pain points of Push mode, makes the client stateless and lighter, and converges the consumption logic to the Broker. Meanwhile, it is friendly to multilingual support. It has also been integrated with Push at the API level, inheriting the simplicity and ease of use of Push API and realizing free switching between Push and POP.
Cloud Future, New Possibilities – Green, Ubiquitous, Trusted Computing
Comprehensive and Deep Integration of Alibaba Cloud Function Compute and EventBridge
495 posts | 48 followers
FollowAlibaba Cloud Native Community - May 16, 2023
Alibaba Cloud Native Community - December 19, 2022
Alibaba Cloud Native Community - October 26, 2023
Alibaba Cloud Native Community - January 5, 2023
Alibaba Cloud Native Community - December 16, 2022
Alibaba Cloud Native Community - March 20, 2023
495 posts | 48 followers
FollowApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn MoreA message service designed for IoT and mobile Internet (MI).
Learn MoreA fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn MoreA distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.
Learn MoreMore Posts by Alibaba Cloud Native Community