Problem description

On the Consumer Status page in the Message Queue for Apache RocketMQ console, the value of real-time message accumulation for group IDs is higher than expected, and the performance is significantly lower.

Cause

Due to the excessive accumulation of messages in the Java process, the real-time message accumulation value of the group ID is higher than expected.

Solution

If the JAVA process has accumulated messages and the performance is degraded, you can take the following measures:

  1. See View Consumer Status to obtain the IP address of the host where messages are accumulated and log on to the host.
  2. Run any of the following commands to view and record the PID of the Java process:
    ps -ef grep java
    jps -lm
  3. Run the following command to view the stack information:
    jstack -l pid > /tmp/pid.jstack
  4. Run the following command to view ConsumeMessageThread information:
    cat /tmp/pid.jstackgrep ConsumeMessageThread -A 10 --color
    The system display is similar to the following. Pay attention to the thread status and stack. Then, see Java documentation to handle the thread status and stack.

References

In addition to the asynchronous decoupling feature, Message Queue for Apache RocketMQ also has an important feature to withstand frontend data peaks to ensure the stability of the backend system. This requires the Message Queue for Apache RocketMQ to have a certain message accumulation capability. Message Queue for Apache RocketMQ can support message accumulation at 1 billion levels without significant performance degradation due to message accumulation. If the performance degradation caused by message accumulation is not resolved, submit a ticket. When you submit a ticket, attach the following information:

  • The heap.bin file.
    1. This file can be obtained by executing the jmap -dump:format=b,file=heap.bin [$PID] command.
      Note: [$PID] is the PID of the JAVA process recorded in Step 2.
    2. Then, run the gzip heap.bin command to generate a compressed package.
  • The local ons.log file of the consumer client where messages are accumulated.
  • The version of the consumer client.

Applicable scope

  • Message Queue for Apache RocketMQ