All Products
Search
Document Center

What should I do if I receive a message accumulation alarm

Last Updated: Mar 28, 2019

Symptoms

  • Log on to the RocketMQ console. Choose Consumers > Consumer Status. Accumulation Amount is set to a larger value.

  • Log on to the RocketMQ console. Choose Message Tracing > Create Query Task > By Message ID. It is found that certain messages have been sent to the broker, but have not been delivered to the downstream consumers.

Analysis

After RocketMQ messages are sent to the broker, the client that is configured with the group ID pulls certain messages from the broker to the local machine for consumption based on the current consumption point. In the consumption process, it may take a long time to consume a single message due to various reasons, such as access to the locked shared resources, competition for the I/O and network resources, and no timer configured for HTTP calls. As a result, messages start to accumulate on the broker.

Solution

Troubleshoot the problem as follows:

  • Log on to the RocketMQ console. Choose Statistics > Message Consumption to query historical consumption records. If message write is faster than message consumption, adjust the code or expand the capacity of the consumer.

  • Print the Jstack information “jstack -l {PID} | grep ConsumeMessageThread” in the application. If messages are blocked, print the Jstack information consecutively five times, and identify the place where the consumption thread is stuck. Rectify the fault, restart the application, and check whether message consumption has recovered.

  • If messages are not accumulating any more, check whether the threshold value is too small and causes message accumulation. Choose Monitoring > Monitoring Items, and click Edit to increase the alarm threshold for message accumulation.

Verification

  • Print the Jstack information “jstack -l {PID} | grep ConsumeMessageThread” in the application. The consumption thread is not blocked.

  • Log on to the RocketMQ console. Choose Consumers > Consumer Status. The value of TPS increases and the value of Accumulation Amount decreases.