edit-icon download-icon

What should I do if I received a message accumulation alarm?

Last Updated: Jul 20, 2018

Description

  • Log on MQ console. Click Consumers > Consumer Status. Accumulation Volume is excessive.

  • Log on MQ console. Click Message Tracing > Create Query Job > By Message ID. Some messages have been sent to broker, but not delivered to consumers.

Analysis

After messages are sent to broker, clients equipped with Consumer ID extract some of the messages according to consumer offset from broker to local for consumption. Access to a shared resource using a lock, IO and network resource conflict, overtime HTTP call, no timeout settings for HTTP request all can cause the slower consumption of a single message and the accumulation of messages.

Solution

Do the followings:

  • Log on MQ console. Click Statistics > Message Consumption. View historical message consumption status. If message is written faster than consumed, adjust code or expand consumers.

  • Print jstack -l {PID} | grep ConsumeMessageThread in application. In case of consumption thread blocking, print Jastck information five times continuously to find which threads are blocked. After fix the blocked threads, restart the application.

  • If messages stop accmulating, check if the threshold value is set too low. Click Monitoring > Monitoring Items. Click Edit to increase the threshold value of the monitoring item.

Result

  • Print jstack -l {PID} | grep ConsumeMessageThread in application. No consumption thread blocking occurs.

  • Log on MQ console. Click Consumers > Consumer Status. Consumption TPS goes up. Accumulation Volume goes down.

Thank you! We've received your feedback.