What do I do if messages are accumulated in a group? - ApsaraMQ for Kafka

When you use ApsaraMQ for Kafka, message accumulation may occur in a group. This topic describes how to determine whether message accumulation is normal and how to handle abnormal message accumulation.

Consumption process on a client

To understand whether message accumulation exists, you must understand how messages are consumed on a client. The following figure describes the consumption process.

Determine whether message accumulation is normal

If message accumulation occurs in a group, perform the following steps to determine whether the message accumulation is normal:

Log on to the ApsaraMQ for Kafka console.
In the top navigation bar, select the region where your ApsaraMQ for Kafka instance resides.
In the left-side navigation pane, click Instances.
On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane of the Instance Details page, click Groups.
On the Groups page, find the group that you want to manage and choose More > Consumer Status in the Actions column.
On the Consumer Status page, determine whether message accumulation is normal based on the values of the Last Consumed At, Accumulated Messages, and Consumer Offset parameters. The values of these parameters are recorded at an interval of 1 minute.
Note
You can click Details to view the consumer offset.
- The most recent consumption time is close to the current time, and the number of accumulated messages fluctuates within an expected stable range. This indicates that the client is pulling new messages, and the message accumulation is normal.
  Note
  The number of accumulated messages varies based on the rate of message production and the frequency to commit consumer offsets. A large value of the Accumulated Messages parameter does not indicate abnormal message accumulation. For example, if the transactions per second (TPS) for sending messages in a topic is 10,000, and the frequency to commit consumer offsets is once per second, the number of accumulated messages may fluctuate around 10,000 in normal scenarios.
- The number of accumulated messages gradually increases, and the consumer offset remains unchanged. This indicates that the consumer thread on the client is blocked, and the client stops consuming messages and committing consumer offsets to the broker. In this case, the message accumulation is abnormal. For information about how to resolve this issue, see the "How to handle abnormal message accumulation" section of this topic.
- The number of accumulated messages gradually increases, and consumer offsets are being committed. This indicates that the client is still consuming messages, but the rate of message consumption is lower than the rate of message production. In this case, the message accumulation is abnormal. Messages are actively pulled from the broker by the client. Therefore, this issue is not caused by operations in Step 1 of the "Consumption process on a client" section of this topic. In this case, check the consumption duration in Step 2. For information about how to resolve this issue, see the "How to handle abnormal message accumulation" section of this topic.
  Note
  In most cases, abnormal message accumulation is caused by slow message consumption or a blocked consumption thread. We recommend that you do not specify long durations for related parameters in the consumption logic.
- Messages are accumulated in partitions, but consumption in the downstream system is normal. The assign consumption mode may be used by the downstream system. In this mode, offsets are managed by yourself. In this case, messages may have been consumed. The messages are displayed as accumulated due to uncommitted offsets. We recommend that you manually commit offsets at the earliest opportunity to resolve the message accumulation issue.

How to handle abnormal message accumulation

After you confirm that abnormal message accumulation exists based on the preceding operations, you can print the message consumption duration or check the execution of the thread based on the stack information. You can use the following methods to increase the message consumption rate to prevent abnormal message accumulation:

Increase the number of consumers. You can add consumers in a process and make sure that each consumer corresponds to one thread. Alternatively, you can deploy multiple consumer processes. If the number of consumers exceeds the number of partitions, the consumption rate cannot be increased. Otherwise, specific consumers become idle.
Increase the number of consumption threads. For more information, see the "Increase consumption rate" section of the Best practices for consumers topic.

Note

You can run the jstack command to query the stack information about Java threads.
If messages are accumulated and the consumer status is abnormal in the ApsaraMQ for Kafka console, rebalances may have occurred. During rebalances, messages are not consumed. In this case, messages are accumulated because a large number of clients connect to or disconnect from ApsaraMQ for Kafka. For more information, see Why do rebalances frequently occur on my consumer client?

References

For information about jstack, see jstack - Stack Trace.
For information about other frequently asked questions (FAQ), see FAQ.