Reclaim MongoDB disk fragments to improve disk usage - ApsaraDB for MongoDB

We recommend the storage analysis feature to reclaim disk fragments. It is simpler and has less service impact. Storage analysis reclaims fragments only from hidden nodes. To reclaim fragments from primary and secondary nodes, first perform a primary/secondary failover.

Storage analysis is available for the following MongoDB versions:

MongoDB 8.0: all minor versions.
MongoDB 7.0: all minor versions.
MongoDB 6.0: all minor versions.
MongoDB 5.0: all minor versions.
MongoDB 4.4: 5.0.7 or later.
MongoDB 4.2: 4.0.23 or later.

Prerequisites

The instance must use the WiredTiger storage engine.

Usage notes

Data backup: Back up your database before reclaiming fragments.
Impact of the compact command:

The compact command rewrites and defragments all data and indexes in a collection to reclaim unused space.
- Read/write blocking and performance impact
  - In versions earlier than MongoDB 4.4, the compact command locks the database and blocks all read and write operations. Heavily fragmented collections may cause the compact command to run for a long time and increase replication latency on hidden nodes. Perform this operation during off-peak hours, increase the oplog size, or upgrade the major version to MongoDB 4.4 or later before reclaiming fragments.
  - In MongoDB 4.4 and later, the compact command does not block read and write operations but can still affect performance. Perform this operation during off-peak hours.
- Node rebuilding
  - For MongoDB 3.4, 4.0, 4.2 (4.0.22 or earlier), or 4.4 (5.0.6 or earlier), a node running compact enters the RECOVERING state. If the operation takes too long, the health check may consider the node unhealthy and trigger node rebuilding. MongoDB minor version release notes.
  - In later MongoDB versions, a node running compact remains in the SECONDARY state and does not trigger node rebuilding.
Ineffective compact command:

The compact command may be ineffective in the following scenarios (block_compact):
- The physical storage size of the collection is less than 1 MB.
- The fragmentation rate is less than 20%.
- Less than 20% of free space exists in the first 80% of the data file, or less than 10% of free space exists in the first 90% of the data file.

Reclamation time: The compact execution time depends on the data volume and system load.
Other notes:
- The space reclaimed by compact may be less than the total free space. Ensure the previous compact operation is complete before starting a new one.
- The compact command can run even when an instance is locked due to full disk space.

Background information

Why do disk fragments occur?

Formation: When data is deleted, the storage space is marked as free. New data may reuse this space or be appended to the end of the file. This creates pockets of unused space known as disk fragments.
Impact: More fragments mean lower effective disk utilization. For example, on a 100 GB disk with 20 GB of fragments and 60 GB of active data, reported utilization is 80% (80 GB/100 GB), but effective utilization is only 60% (60 GB/100 GB).

When to reclaim disk fragments

After you delete a large volume of data

When you delete a large number of documents, the freed space is not returned to the operating system but reserved for future writes. This can leave significant fragmented space on the disk.

Important
Neither manual deletion nor TTL expiration automatically reclaims disk fragments. You must reclaim the space manually.
After prolonged high-write workloads

Sustained high-write workloads (frequent inserts, updates, and deletes) gradually accumulate fragmented space across the disk.
When disk space is low and fragmentation exceeds 20%

When disk utilization reaches 85% to 90% or higher, reclaiming fragments can free up space and reduce storage pressure.

View disk storage space

View collection storage status

Run the db.runCommand({collStats: <collection_name>}) command to view storage status. Key output fields:

size: The logical storage size of the collection.
storageSize: The physical storage size of the collection.
freeStorageSize: The reclaimable free space within the collection. Available in MongoDB 4.4 and later.

After deleting documents with the remove command, size decreases but storageSize may not. A high freeStorageSize to storageSize ratio indicates high fragmentation.

Note

These fields (size, storageSize, freeStorageSize) are documented in collStats-Output.

Estimate reclaimable space

Connect to your ApsaraDB for MongoDB instance using the mongo shell. For replica set instances, connect to a secondary node to minimize service impact. Connection instructions:
Switch to the database that contains the collection.
Syntax:
```
use <database_name>
```
Parameter description: <database_name> is the name of the database that contains the collection.

Note
You can run the show dbs command to query the existing databases.

Example:
Switch to the test_database database.
```
use test_database
```
Check the amount of reclaimable disk fragment space for the collection.
Syntax:
```
db.<collection_name>.stats().wiredTiger["block-manager"]["file bytes available for reuse"]
```
Parameter description: <collection_name> is the name of the collection.

Note
You can run the show tables command to query the existing collections.
Example:
```
db.test_database_collection.stats().wiredTiger["block-manager"]["file bytes available for reuse"]
```
The following result is returned:
```
207806464
```
This output indicates that the estimated reclaimable space is 207,806,464 bytes.

Reclaim disk fragments

Standalone or replica set instances

A standalone instance has a single node. Connect to it and run the compact command to reclaim disk fragments.
A replica set instance has multiple nodes. Reclaim fragments from both the primary and secondary nodes.
Important
- To minimize service impact, reclaim fragments from secondary nodes first. Then perform a primary/secondary failover and reclaim fragments from the former primary node (now secondary).
- If the replica set instance has read-only nodes, reclaim their disk fragments as well. The command is the same as for primary and secondary nodes.

Connect to the instance using the mongo shell. Connection instructions:
- Connect to a standalone instance using the mongo shell
- Connect to a replica set instance using the mongo shell
Switch to the database that contains the collection.
Syntax:
```
use <database_name>
```
Parameter description: <database_name> is the name of the database that contains the collection.

Note
You can run the show dbs command to query the existing databases.

Example:
Switch to the replica_database database.
```
use replica_database
```
View the disk space occupied by the database before you reclaim disk fragments.
```
db.stats()
```
Note
You can copy and run this command without modification.
Reclaim disk fragments from the collection.
Syntax:
```
db.runCommand({compact:"<collection_name>",force:true})
```
Parameters:
- <collection_name>: The name of the collection.
  
  Note
  You can run the show tables command to query the existing collections.
- force: Optional. The value must be true.
  
  This parameter is required if you run this command on the primary node of an ApsaraDB for MongoDB instance running version 4.2 or earlier.
Example:
```
db.runCommand({compact:"sharded_collection"})
```
A successful operation returns the following result:
```
{ "ok" : 1 }
```
View the disk space occupied by the database after reclaiming disk fragments.
```
db.stats()
```
Note
You can copy and run this command without modification.

Sharded cluster instances

For a sharded cluster instance, reclaim disk fragments only from the shard nodes. The mongos and config server components do not store user data, so reclaiming their fragments is unnecessary.

Note

The compact command is not supported on read-only nodes of a sharded cluster instance.

Connect to the sharded cluster instance using the mongo shell. Connect to a sharded cluster instance using the mongo shell.
Switch to the database that contains the collection.
Syntax:
```
use <database_name>
```
Parameter description: <database_name> is the name of the database that contains the collection.

Note
You can run the show dbs command to query the existing databases.

Example:
Switch to the sharded_database database.
```
use sharded_database
```
View the disk space occupied by the database before you reclaim disk fragments.
```
db.stats()
```
Note
You can copy and run this command without modification.
Reclaim disk fragments from the collection.
Reclaim fragments from both the primary and secondary nodes in each shard.

Important
To minimize service impact, reclaim fragments from secondary nodes first. Then perform a primary/secondary failover and reclaim fragments from the former primary node (now secondary).
- Reclaim disk fragments from the secondary nodes in the shard component.
  
  The command differs for mongo shell and mongosh. Select the tab that corresponds to your client.
  
  Note
  Mongosh 2.x adds a parameter to specify the read preference, unlike mongosh 1.x.
  Mongo shell
  
  Syntax:
  db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>"},$queryOptions: {$readPreference: {mode: 'secondary'}}})
  
  Parameters:
  
  <Shard ID>: The ID of the shard component.
  
  Note
  Log on to the MongoDB console. On the Basic Information page of the target instance, find the ID of the shard component in the Shard List section.
  
  <collection_name>: The name of the collection.
  
  Note
  You can run the show tables command to query the existing collections.
  
  Example:
  db.runCommand({runCommandOnShard:"shard01","command":{compact:"sharded_collection"},$queryOptions: {$readPreference: {mode: 'secondary'}}})
  Mongosh 1.x
  
  Syntax:
  db.getMongo().setReadPref('secondary') db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>"}})
  
  Parameters:
  
  <Shard ID>: The ID of the shard component.
  
  Note
  Log on to the MongoDB console. On the Basic Information page of the target instance, find the ID of the shard component in the Shard List section.
  
  <collection_name>: The name of the collection.
  
  Note
  You can run the show tables command to query the existing collections.
  
  Example:
  db.getMongo().setReadPref('secondary') db.runCommand({runCommandOnShard:"d-2ze91ae9d55d6604","command":{compact:"test"}})
  Mongosh 2.x
  
  Syntax:
  db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>"}},{readPreference: "secondary"})
  
  Parameters:
  
  <Shard ID>: The ID of the shard component.
  
  Note
  Log on to the MongoDB console. On the Basic Information page of the target instance, find the ID of the shard component in the Shard List section.
  
  <collection_name>: The name of the collection.
  
  Note
  You can run the show tables command to query the existing collections.
  
  Example:
  db.runCommand({runCommandOnShard:"d-2ze657bce53fb6d4","command":{compact:"test_collection"}}, { readPreference: "secondary" })
- Reclaim disk fragments from the primary node in the shard component.
  Syntax:
```
db.runCommand({runCommandOnShard:"<Shard ID>","command":{compact:"<collection_name>",force:true}})
```
  Parameters:
  - <Shard ID>: The ID of the shard component.
    
    Note
    Log on to the MongoDB console. On the Basic Information page of the target instance, find the ID of the shard component in the Shard List section.
  - <collection_name>: The name of the collection.
    
    Note
    You can run the show tables command to query the existing collections.
  - force: Optional. The value must be true.
    
    This parameter is required if your sharded cluster instance is running version 4.2 or earlier.
  Example:
```
db.runCommand({runCommandOnShard:"shard01","command":{compact:"sharded_collection",force:true}})
```
View the disk space occupied by the database after reclaiming disk fragments.
```
db.stats()
```
Note
You can copy and run this command without modification.

FAQ

Q: The command fails with "Compaction interrupted on table:xxx due to cache eviction pressure' on server xxx."

A: Older, small-spec instances may exit prematurely during compact due to cache pressure. Perform this operation during off-peak hours.