ApsaraDB for MongoDB:What do I do if my instance is locked or data cannot be written to the instance due to exhausted disk space? - ApsaraDB for MongoDB

When an ApsaraDB for MongoDB instance runs out of disk space, the instance enters a Locked state that blocks both writes and deletes. This topic explains how to confirm the root cause and resolve the issue.

When disk space is exhausted on a sharded cluster instance, the instance does not enter the Locked state.

Symptoms

Your application can read data but cannot write data.

Connecting via mongo shell and attempting a write returns error code 13 (Unauthorized):

db.customer.insert({"name":"zhangsan"})
WriteCommandError({
        "operationTime" : Timestamp(1563437183, 1),
        "ok" : 0,
        "errmsg" : "not authorized on db1 to execute command { insert: \"customer\", ordered: true, lsid: { id: UUID(\"8d43461c-5c51-49ef-b9b3-9xxxxxxxxf\") }, $clusterTime: { clusterTime: Timestamp(1563437183, 1), signature: { hash: BinData(0, 0C3FAAE747xxxxxx), keyId: 668293399xxxxxx } }, $db: \"db1\" }",
        "code" : 13,
        "codeName" : "Unauthorized",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1563437183, 1),
                "signature" : {
                        "hash" : BinData(0,"DD+q50dPTuIQKTzytT5SiTPYX4Q="),
                        "keyId" : NumberLong("66xxxxxxxx")
                }
        }
})

The instance status shows Locked in the ApsaraDB for MongoDB console.

Check disk space usage

The system collects disk space usage data every five minutes.

Log on to the ApsaraDB for MongoDB console.
In the upper-left corner, select the resource group and region of the instance.
In the left-side navigation pane, click Replica set instances or Sharded cluster instance based on your instance type.
Find and click the ID of the instance.
Check disk usage based on instance type:
- Standalone or replica set instances On the Basic Information page, check the instance status and disk space usage. If the status is Locked and disk usage exceeds 100%, disk space is exhausted.
- Sharded cluster instances
  1. In the left-side navigation pane, click Monitoring Data.
  2. On the Monitoring Data page, select a shard node to monitor.
    Node IDs starting with d are shard nodes. Node IDs starting with s are mongos nodes.
  3. Check disk space usage. If usage exceeds 100% on any shard node, disk space is exhausted.

Solutions

Expand disk space

Expand the disk space by modifying the instance configurations. For details, see Overview of changing instance configurations.

The maximum disk space available depends on instance specifications. For details, see Instance types overview.

Migrate to a sharded cluster instance

A replica set instance supports up to 3,000 GB of disk space. If you need more space, migrate to a sharded cluster instance, which supports up to 96,000 GB by adding shards. Use Data Transmission Service (DTS) to migrate. For details, see Migrate data from a replica set instance to a sharded cluster instance.

Optimization suggestions

If you recently deleted large amounts of data using db.collection.remove or have not run defragmentation, disk fragments may be consuming space that MongoDB cannot reuse for new writes.

To reclaim fragmented disk space, see Defragment the disks of an instance to increase disk utilization.