After Message Queue for Apache RocketMQ messages are sent to the broker, the client that is configured with the group ID pulls certain messages from the broker to the local machine for consumption based on the current consumption offset. In the consumption process, it may take a long time to consume a single message due to various reasons, such as access to the locked shared resources, competition for the I/O and network resources, and no timer being configured for HTTP calls. As a result, messages start to accumulate on the broker.

Symptoms

  • Log on to the Message Queue for Apache RocketMQ console. Choose Groups > Consumer Status. Accumulation Amount is set to a value higher than the expected one.
  • Log on to the Message Queue for Apache RocketMQ console. Choose Message Tracing > Create Query Task > By Message ID. It is found that certain messages have been sent to the broker, but have not been delivered to the downstream consumers.

Analysis

After Message Queue for Apache RocketMQ messages are sent to the broker, the client that is configured with the group ID pulls certain messages from the broker to the local machine for consumption based on the current consumption offset. In the consumption process, it may take a long time to consume a single message due to various reasons, such as access to the locked shared resources, competition for the I/O and network resources, and no timer being configured for HTTP calls. As a result, messages start to accumulate on the broker.

Solution

Perform the following steps:

  • Log on to the Message Queue for Apache RocketMQ console. Choose Statistics > Message Consumption to query historical consumption records. If message writing is faster than message consumption, modify the code or increase the capacity of the consumer.
  • Print the JStack information "jstack -l {pid} | grep ConsumeMessageThread" in the application. If messages are blocked, print the JStack information consecutively five times, and identify the place where the consumption thread is stuck. Rectify the fault, restart the application, and check whether message consumption has recovered.
  • If messages are not accumulated any more, check whether the threshold value is too small and causes message accumulation. Choose Monitoring and then click Edit to increase the alert threshold for message accumulation.

Verification

  • Print the JStack information jstack -l {pid} | grep ConsumeMessageThread in the application. The consumption thread is not blocked.
  • Log on to the Message Queue for Apache RocketMQ console. Choose Groups > Consumer Status. The value of TPS is increased and the value of Accumulation Amount is decreased.