ApsaraDB for MongoDB:Troubleshoot high memory usage on an ApsaraDB for MongoDB instance

For a replica set instance, you can use the following methods to view the memory usage:

• View memory usage in monitoring charts

  A replica set instance consists of multiple node roles. Each node role can correspond to one or more physical nodes. A replica set instance consists of a primary node that supports read and write operations, one or more high-availability secondary nodes, a hidden node, and one or more optional read-only nodes.

  On the Monitoring Info page of an instance in the ApsaraDB for MongoDB console, view the memory usage on the corresponding node in monitoring charts.

• View IOPS usage by running commands

  To view and analyze the memory usage on an instance, run the db.serverStatus().mem command in the mongo shell. A response similar to the following one is returned:

  { "bits" : 64, "resident" : 13116, "virtual" : 20706, "supported" : true }
  //resident indicates the physical memory that is consumed by the mongod process. Unit: MB. 
  //virtual indicates the virtual memory that is consumed by the mongod process. Unit: MB. 

  Note

  For more information about serverStatus, see serverStatus.

You can use the following methods to view the memory usage of the engine:

• Run the db.serverStatus().wiredTiger.cache command in the mongo shell. The bytes currently in the cache value is the memory size. Sample result:

  {
     ......
     "bytes belonging to page images in the cache":6511653424,
     "bytes belonging to the cache overflow table in the cache":65289,
     "bytes currently in the cache":8563140208,
     "bytes dirty in the cache cumulative":NumberLong("369249096605399"),
     ......
  }

• On the Dashboard page of an instance in the DAS console, view the percentage of dirty data in the WiredTiger cache. For more information, see Performance trends.

• Use the mongostat tool of ApsaraDB for MongoDB to check the percentage of dirty data in the WiredTiger cache. For more information, see mongostat.

If a large number of connections are connected to the instance, the memory may be consumed due to the following reasons:

• Each connection has a corresponding thread to process the request in the background. Each thread can use up to 1 MB of stack space. In most cases, the overhead is dozens to hundreds of KB.

• Each TCP connection has read and write buffers at the kernel layer. The buffer size is determined by TCP kernel parameters such as tcp_rmem and tcp_wmem. You do not need to specify the buffer size. However, more concurrent connections consume larger amounts of socket cache space and results in higher memory usage of TCP.

• Each request has a unique context. Multiple temporary buffers may be allocated for request packets, response packets, and ordering. The temporary buffers are gradually released at the end of each request. The buffers are first released to the TCMalloc cache and then gradually released to the operating system.

  In many cases, memory usage is high because TCMalloc fails to promptly release the memory that requests consume. Requests may consume up to dozens of GB of memory before the memory is released to the operating system. To query the size of memory that TCMalloc has not released to the operating system, run the db.serverStatus().tcmalloc command. The TCMalloc cache size is the sum of the pageheap_free_bytes and total_free_byte values. Sample result:

  {
     "generic":{
             "current_allocated_bytes":NumberLong("9641570544"),
             "heap_size":NumberLong("19458379776")
     },
     "tcmalloc":{
             "pageheap_free_bytes":NumberLong("3048677376"),
             "pageheap_unmapped_bytes":NumberLong("544994184"),
             "current_total_thread_cache_bytes":95717224,
             "total_free_byte":NumberLong(1318185960),
  ......
     }
  }

  Note

  For more information about TCMalloc, see tcmalloc.

In normal data writes, secondary nodes maintain a buffer of about 256 MB for data replay. After the primary node creates indexes, secondary nodes may consume more memory for data replay. In versions earlier than MongoDB 4.2, indexes are created in the background on the primary node. Serial replay for index creation may consume a maximum of 500 MB memory. In MongoDB 4.2 and later, indexes cannot be created in the background and secondary nodes can perform parallel replay for index creation. This requires more memory because instance OOM errors may occur when multiple indexes are created at a time.

The goal of memory optimization is not to minimize memory usage. Instead, memory optimization seeks a balance between resource consumption and performance. Ideally, the memory remains sufficient and stable and system performance is not affected. You cannot change the cachesize value that ApsaraDB for MongoDB configures. We recommend that you use the following methods to optimize memory usage:

• Control the number of concurrent connections. 100 persistent connections can be created in a database based on the results of performance tests. By default, a MongoDB driver can establish 100 connection pools with the backend. If a large number of clients exist, you must reduce the size of the connection pool for each client. We recommend that you establish no more than 1,000 persistent connections in a database. Otherwise, overheads in the memory and multi-thread context may increase and result in processing latency for requests.

• Reduce the memory overhead of a single request. For example, you can create indexes to reduce the number of collection scans and perform memory ordering.

• If the number of connections is appropriate but the memory usage continues to increase, we recommend that you upgrade the memory configurations. Otherwise, system performance may sharply decline due to OOM errors and extensive cache clearing.

• In scenarios where memory leaks may occur, contact Alibaba Cloud technical support.