All Products
Search
Document Center

Insufficient application memory

Last Updated: Mar 28, 2019

Symptoms

  • The memory is exhausted on the machine where the application is deployed.

  • The keyword OutOfMemory can be found in /{user.home}/logs/ons.log.

  • According to the user-side feedback, in the RocketMQ console, choose Consumers > Consumer Status.Accumulation Amount shows that many messages have accumulated, and Connection Status shows that message accumulation has occurred on all the connected clients. Troubleshooting through Jstack shows that the ConsumeMessageThread _ thread is not blocked.

Analysis

A RocketMQ consumer actively pulls messages from the broker, caches them to the client, and then delivers the messages to the business consumption logic.

In versions earlier than 1.7.0. Final, the client caches a maximum of 1000 messages to each queue of each topic by default. Assume that each topic has 16 queues (two primary clusters and two standby clusters, and eight queues on each broker) and that the average size of messages ordered by this topic is 64 KB. The size of the messages cached by this topic on the client is 16 * 1000 * 64 KB = 1 GB. If the user subscribes to eight topics at the same time and caches messages in the client memory, the memory usage will eventually exceed the user’s JVM configuration, resulting in OOM.

Cause 1

Depending on ons-client versions earlier than 1.7.0. Final, the average size of messages for each topic exceeds 4 KB. In addition, message consumption is slow, which easily leads to cache of messages in the client memory.

Verification method

Check whether the keyword OutOfMemory can be found in /{user.home}/logs/ons.log, or run the command jmap -dump:live,format=b,file=heap.bin <pid> to verify the objects that occupy a large amount of memory.

Recovery solution

Upgrade ons-client to 1.7.0. Final or later, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMess ageSizeInMiB parameter for the corresponding ConsumerBean. Then, restart the application.

Cause 2

Depending on ons-client 1.7.0. Final or later, the maximum memory consumption is 512 MB by default (that is, the total cache size of all topics subscribed by a group ID). If the application still has the OOM problem, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMessageSizeInMiB parameter to customize the maximum memory consumption (ranging from 16 MB~2,048 MB) upon startup of ConsumerBean.

Verification method

Check the ons-client version used by the application and use the JVM to confirm the size of memory allocated to the process.

Recovery solution

Based on the memory usage of the machine where the application runs, configure the com.aliyun.openservices.ons.api.PropertyKeyConst#MaxCachedMess ageSizeInMiB parameter for the corresponding ConsumerBean. Then, restart the application.

Verification

  • The keyword OutOfMemory cannot be found in /{user.home}/logs/ons.log.

  • Log on to the RocketMQ console. Choose Consumers > Consumer Status. The value of TPS increases and the value of Accumulation Amount decreases.