All Products
Search
Document Center

Write failures caused by disk space exhaustion in MongoDB

Last Updated: Sep 24, 2020

Problem description 

If the disk space of an ApsaraDB for MongoDB instance is exhausted, the instance is locked and you cannot write data to or delete data from it. The description of the problem scenario complies with the following requirements:

  • The deployed application cannot write data to the database, but can read the data normally.
  • When you connect to a database through the Mongo Shell for troubleshooting, the following error message is returned after a data record is written.
    not authorized on xxxx to execute command
  • In the apsaradb for MongoDB console, you can view the status of an instance in the locked.
    Description due to special features of the sharded cluster architecture, when the disk space of a sharded cluster instance is fully occupied, the instance is not displayed as locked.

Cause

Disk space run out.

Solution

Alibaba Cloud reminds you that:

  • Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
  • You can modify the configurations and data of instances including but not limited to Elastic Compute Service (ECS) and Relational Database Service (RDS) instances. Before the modification, we recommend that you create snapshots or enable RDS log backup.
  • If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify such information in a timely manner.

If you cannot write data to a disk of an apsaradb for MongoDB instance, perform the following steps for troubleshooting.

Troubleshooting

  1. Log on to the ApsaraDB for MongoDB console.
  2. In the top navigation bar, select the region where the target RDS instance resides
  3. In the left-side navigation pane, click replica Set instances or sharded cluster instances.
  4. Click the ID of the target instance.
  5. Perform operations based on the instance type by referring to the following content:
    Description: the data collection granularity of disk space usage is 5 minutes.
    • Standalone or replica set instance
      In the basic information page to check the instance status and disk space usage. Confirm that the instance status is locked at the same time, check that the disk space usage of the instance exceeds 100%.
    • Sharded cluster instance
      1. In the left-side navigation pane, choose monitoring information.
      2. In monitoring information at the top of the page, filter the Shard you want to view.
        Description: used to D the ID of the node that starts with a letter is a Shard node. s the node ID that starts with: Mongos.
      3. Check the disk space usage of the Shard. Make sure that the disk space usage of the Shard exceeds 100%.

Problem handling

To fully occupy the disk space, you can use the following two methods:

  1. Improve instance disk space by changing the configuration Solution Overview. For more information, see configuration change Overview.
    Description: The maximum disk space varies with instance types. For more information, see instance specifications.
  2. A replica set instance supports up to 3,000 GB of disk space. If you need more services, we recommend that you use a sharded cluster instance where you can scale out the disk space by adding shards. The maximum size is 96,000GB. For more information, see sharded cluster instances.
    Description: you can use data transmission service (DTS) to migrate data from the source instance to the destination instance. For more information, see migrate data from a replica set instance to a sharded cluster instance.

References

If your instance uses db.collection.remove command deleted a large amount of data or no defragmentation has been performed. For more information, see organize physical space debris to improve disk usage.

Application scope

  • ApsaraDB for MongoDB