All Products
Search
Document Center

Message accumulation occurs when using Message Queue for Apache Kafka

Last Updated: Sep 02, 2021

Overview

When you use Message Queue for Apache Kafka, you may find message accumulation in the group. For more information about whether the situation is normal and how to handle it, see the following.

Background information

Client consumption process

Understanding the client consumption process helps to understand whether messages are accumulated. In general, the client consumption process is roughly as follows:

Step 1 Step 2 Step 3

consumer calls the server interface to pull messages.

After the message is pulled, the client consumption logic is executed.

After successful consumption, if the offset submission interval is exceeded, submit the offset to the server. If the offset submission interval is not exceeded, submit the offset after the next consumption is completed. Generally, an interval of 1 to 2 seconds is recommended.

Determine whether message accumulation is normal.

If message accumulation is found in a group, see the following steps to determine whether message accumulation is normal:

  1. Log on to the Message Queue for Apache Kafka console.
  2. In the top navigation bar, select a region.
  3. In the left-side navigation pane, click Instances.
  4. On the Instances page, click the name of the target instance.
  5. On the Instance Details page, click Groups in the left-side navigation pane.
  6. On the Groups page, find the target Group and choose More > Consumption Status in the Actions column.
  7. On the Consumption Status page, record the latest consumption time, accumulation amount, and message offset of each topic partition at an interval of 1 minute, and then perform the following judgment:
    Note: You can click Details to view the message offset.
    • The latest consumption time is close to the current time, and the accumulation amount keeps fluctuating between a stable value without continuous expansion.
      This indicates that the client has been pulling the latest messages and no messages are accumulated. This is normal.
      Note: The amount of message accumulation varies based on the number of messages and the submission frequency of consumer. A large value does not necessarily indicate accumulation. For example, the sending frequency of a topic is 10000tps, and the submission frequency of a consumer point is 1s. Under normal circumstances, the displayed accumulation amount may fluctuate around 10000, but it does not mean that messages are accumulated.

    • The accumulation volume gradually expands, and the message offset remains unchanged.
      The consumer thread of the client is stuck due to some reasons and does not continue to consume or submit the offset to the server. This is an exception, that is, messages are accumulated. For more information, see How to handle message accumulation.
    • Accumulation is gradually expanding, while consumption sites are advancing.
      indicates that the client is still consuming, but the message consumption speed is slower than the message sending speed. Because the client proactively pulls messages from the server, Step 1 in the client consumption process generally does not cause the consumption bottleneck. To troubleshoot the consumption time in Step 2, see How to handle message accumulation.
      Note: Message accumulation is mostly caused by slow consumption or blocked consumption threads. We recommend that you do not use too many time-consuming operations in the consumption logic.

How to handle message accumulation

After the preceding judgment, it is recommended to print the consumption time of the message, or view the thread execution status based on the stack information, and adjust it appropriately to speed up the message consumption and avoid message accumulation.

Note :Java processes can use Jstack to print the stack information of consumer processes.

References

Applicable scope

  • Message Queue for Apache Kafka