Symptoms

  • The memory is exhausted on the machine where the application is deployed.
  • The keyword OutOfMemory can be found in /{user.home}/logs/ons.log.
  • In the Message Queue for Apache RocketMQ console, choose Groups > Consumer Status. Accumulation Amount shows that many messages have accumulated, and Connection Status shows the message accumulation on each client. Troubleshooting through JStack shows that the ConsumeMessageThread _ thread is not blocked.

Analysis

In versions earlier than 1.7.0.Final, the client caches a maximum of 1000 messages to each queue of each topic by default. Assume that each topic has 16 queues (two primary clusters and two secondary clusters, and eight queues on each broker) and that the average size of messages under this topic is 64 KB. The size of the messages cached for this topic on the client is 16 x 1000 x 64 KB = 1 GB. If the user subscribes to eight topics at the same time and caches messages in the client memory, the memory usage will eventually exceed the user's Java Virtual Machine (JVM) configuration, resulting in out of memory (OOM).

  • Cause 1

    The ons-client version is earlier than 1.7.0.Final, and the average size of messages under the topic exceeds 4 KB. In addition, message consumption is slow, which easily leads to cache of messages in the client memory.

    Verification method

    Check whether the keyword OutOfMemory can be found in /{user.home}/logs/ons.log, or run jmap -dump:live,format=b,file=heap.bin <pid> to check the objects that occupy a large amount of memory.

    Solution

    Upgrade ons-client to 1.7.0.Final or later, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMess ageSizeInMiB parameter for the corresponding ConsumerBean. Then, restart the application.

  • Cause 2

    For ons-client 1.7.0.Final or later, the maximum memory consumption is 512 MB by default, which is the total cache capacity of all topics subscribed to a group ID. If the application still suffers OOM, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMessageSizeInMiB parameter to customize the maximum memory consumption (ranging from 16 MB to 2,048 MB) upon startup of ConsumerBean.

    Verification method

    Check the ons-client version used by the application and check the corresponding JVM parameter value to confirm the size of memory allocated to the process.

    Solution

    Based on the memory usage of the machine where the application runs, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMessageSizeInMiB parameter for the corresponding ConsumerBean. Then, restart the application.

Verification

  • The keyword OutOfMemory cannot be found in /{user.home}/logs/ons.log.
  • Log on to the Message Queue for Apache RocketMQ console. Choose Groups > Consumer Status. The value of TPS increases and the value of Accumulation Amount decreases.