ApsaraDB for MongoDB:High disk space utilization in MongoDB instances

Sharded cluster architecture

If your ApsaraDB for MongoDB instance uses a sharded cluster architecture, log on to the ApsaraDB for MongoDB console and use the following methods to check its space usage:

• Monitoring chart analysis

  On the Monitoring Information page, select a target node to view its Disk Usage (Bytes) and Disk Usage (%).

• Detailed analysis

  Sequentially analyze the space usage of each node by using MongoDB's native commands db.stats() and db.$collection_name.stats().

The compact method

To minimize impact on your business, first run the db.runCommand({compact:"collectionName"}) command on a secondary node, and then perform a primary/secondary switchover. Replace collectionName with the name of your collection.

For more information about how to use the compact command, see Defragment disks to improve disk utilization.

Space gap from large journal logs

In ApsaraDB for MongoDB versions earlier than 4.0, if the number of open files on the host machine reaches its limit, the internal log server cleanup thread is interrupted and the journal log grows indefinitely. If you see log entries similar to the following in the MongoDB runtime logs, upgrade the kernel to ApsaraDB for MongoDB 4.0 or later. As a temporary fix, restart the mongod process. For more details, see log-server thread exit quietly on error while the mongodb process still running.

2019-08-25T09:45:16.867+0800 I NETWORK [thread1] Listener: accept() returns -1 Too many open files in system 
2019-08-25T09:45:17.000+0800 I - [ftdc] Assertion: 13538:couldn't open [/proc/55692/stat] Too many open files in system src/mongo/util/processinfo_linux.cpp 74
2019-08-25T09:45:17.002+0800 W FTDC [ftdc] Uncaught exception in 'Location13538: couldn't open [/proc/55692/stat] Too many open files in system' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.

Inappropriate shard key type

In a sharded cluster, the choice of shard key type is critical. The two main types are hashed sharding and ranged sharding. For disk balancing, hashed sharding is often a better strategy than ranged sharding because ApsaraDB for MongoDB uses an internal hash function to evenly distribute data across shards based on key values. In contrast, ranged sharding distributes data based on key value ranges, which can concentrate new data in a "hot" chunk, causing high disk I/O on that shard and short-term data imbalance.

Inappropriate shard key field

Although the number of chunks on each shard may be similar, most of the data is concentrated in a few chunks. As a result, the shards containing these "hot" chunks store significantly more data than others. When you run the sh.status() command and check the ApsaraDB for MongoDB runtime logs, you may see warning messages similar to the following:

2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260000" } 
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260200" } 
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260230" }

The mongos balancer primarily considers chunk counts on each shard to determine whether data is balanced. This can lead to a scenario where chunk counts are balanced but actual data distribution is heavily skewed. This happens because shard key values within a chunk are almost identical. When the chunk reaches the 64 MB split threshold, MongoDB splits it and creates an empty chunk. Over time, although chunk counts increase and migrations occur, the migrated chunks are often empty—resulting in balanced chunk counts but imbalanced data. To fix this, redesign your sharding strategy to use a high-cardinality field as the shard key.

Unsharded databases in a sharded cluster

An ApsaraDB for MongoDB sharded cluster instance supports both sharded and unsharded databases. All data from an unsharded database resides on a single primary shard, so if that database is large, its primary shard may store significantly more data than other shards.

This issue can also occur when you import data from a source mongos cluster to a new one but neglect to configure sharding on the destination cluster before the import.

To address these issues, we recommend the following:

• If you are initializing a destination cluster with an import, design and enable sharding before you start.

• If you have many unsharded databases of roughly equal size, use the movePrimary command to move specific databases to different shards.

• If a very large unsharded database exists, we recommend either sharding it or migrating it to a standalone replica set instance.

• If this situation occurs but you have sufficient disk space, you can choose to ignore the issue.