ApsaraDB for MongoDB:Troubleshoot high disk utilization issues on an instance

Journals growing without bound (MongoDB earlier than 4.0)

In MongoDB versions earlier than 4.0, if the number of open files on the host reaches the system limit, the cleaner threads on the MongoDB log server exit silently. Journal files then grow without limit.

Look for entries like the following in the instance's runtime logs:

2019-08-25T09:45:16.867+0800 I NETWORK [thread1] Listener: accept() returns -1 Too many open files in system
2019-08-25T09:45:17.000+0800 I - [ftdc] Assertion: 13538:couldn't open [/proc/55692/stat] Too many open files in system src/mongo/util/processinfo_linux.cpp 74
2019-08-25T09:45:17.002+0800 W FTDC [ftdc] Uncaught exception in 'Location13538: couldn't open [/proc/55692/stat] Too many open files in system' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.

Resolve

Upgrade MongoDB to 4.0 or later. As a temporary measure, restart the mongod process. See the upstream bug report: WT-4083.

Verify

After upgrading or restarting, confirm that journal files are no longer growing by checking Disk Usage (Bytes) in Monitoring Data over a 10–15 minute window.

Oplog consuming growing space after replication lag or physical backup

Two scenarios cause oplog space to expand and not shrink automatically:

• Replication lag: When secondary nodes fall behind, the available oplog space is no longer capped by the fixed collection size in the configuration file. It can reach up to 20% of the disk space provisioned for the instance. After the lag clears, the physical space is not automatically released.

• Physical backup on a hidden node: A large number of checkpoints are generated during backup, producing substantial log data.

Resolve

Run compact on the oplog collection:

db.grantRolesToUser("root", [{db: "local", role: "dbAdmin"}])
use local
db.runCommand({ compact: "oplog.rs", force: true })

Verify

After the operation, check Disk Usage (Bytes) in Monitoring Data to confirm that log space has decreased on the affected node.

Poor shard key choice: low cardinality

If most data ends up in a small number of chunks—while chunk count across shards stays balanced—the root cause is a low-cardinality shard key.

When a shard key has very few distinct values, the balancer can split and migrate chunks but cannot split a chunk whose documents all share the same key value. The result: chunk counts look balanced, but data sizes are heavily skewed.

Look for warnings in the output of sh.status():

2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260000" }
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260200" }
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260230" }

Resolve

Redesign the shard key using a field with high cardinality. Consider hashed sharding, which distributes data evenly by applying a hash function to shard key values. Ranged sharding distributes data by value range, which tends to concentrate writes on a single chunk. See shard key concepts, hashed sharding, and ranged sharding.

When a chunk reaches 64 MB, MongoDB creates a new empty chunk so migration can continue. If chunks are balanced but data sizes differ greatly across shards, a low-cardinality shard key is the likely cause.

Unsharded databases creating a jumbo shard

Data in an unsharded database is stored entirely on one shard. If that database is large, one shard ends up holding significantly more data than the others. The same situation can occur when data is imported into a sharded cluster instance that was not sharded before the import.

Resolve

Choose the appropriate action based on your situation:

For more on how chunks are partitioned and split, see Data partitioning with chunks and Split chunks in a sharded cluster.

Disk fragmentation from moveChunk operations

When the balancer migrates a chunk, it removes source documents after writing them to the destination. By default, that removal does not release the physical disk space—the data files and index files for the WiredTiger engine retain the space until explicitly reclaimed. This is common when sharding is added to an instance that has been running for some time.

Resolve

Run compact on each affected shard to reclaim fragmented space:

db.runCommand({compact: "<collectionName>"})

See Migrate ranges in a sharded cluster and Manage sharded cluster balancer for context on moveChunk behavior.

Verify

After compact completes on each shard, compare the Disk Usage (Bytes) values across shards in Monitoring Data to confirm the distribution is more even.