All Products
Search
Document Center

Elastic Compute Service:How to troubleshoot the problem of full disk space and inode space on Linux instances

Last Updated:Jan 28, 2021

Disclaimer: this article may be contributed by the community or involve third-party product information, we recommend that you visit the community or the official website of third-party products for help and support. Third-party products will not be provided by Alibaba Cloud after-sales support. This document is for reference only. Alibaba Cloud makes no guarantee by express or any other means.

Problem description

When you create files on an Linux ECS instance, a message similar to the following is prompted:

No space left on device...

Cause

The possible causes of this problem are as follows:

  • The disk partition usage is 100%.
  • The inode usage of the disk partition reaches.
  • Zombie files exist.
    Note: deleted files are not released because the handle is occupied. Therefore, the corresponding space is not released.
  • The name of the Mount point to be overwritten.
    Note: a large number of files already exist in the corresponding directory of the original file system. After the new disk is attached, df command can count the usage of the relevant space, su command does not count.

Solution

Alibaba Cloud reminds you that:

  • Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
  • If you modify the configurations and data of instances including but not limited to ECS and RDS instances, we recommend that you create snapshots or enable RDS log backup.
  • If you have authorized or submitted security information such as the logon account and password in the Alibaba Cloud Management console, we recommend that you modify such information in a timely manner.

To solve the problem, use the following methods based on different causes:

Processing of partition capacity full

  1. Log on to the server and run the df-h command to view the disk usage. Mountedon indicates the directory to which the disk is mounted.

  2. Go to the root directory and run the following command to check the directory that occupies the most disk space level by level until the most accurate file or directory is found.
    du -sh *
    The following command output is returned.

  3. Finally, determine whether to delete the relevant files or directories based on business conditions, or purchase a larger data disk to share the processing.

Processing of inode capacity full

To solve the problem of inode capacity full, follow these steps:

Query inode usage

The inode node of Linux records important information such as file type, size, permissions, owner, number of file connections, creation time, and update time. Another important part is the pointer pointing to the data block. In general, no special configuration is required. If many files are stored, configuration is required. Sometimes there are surplus disk space that cannot store files, which may be caused by the exhaustion of inode.

  1. You can run the df-i command to query the inode usage.

  2. If the inode usage reaches or is close to 100%, you can use either of the following methods:

Clears files or directories that occupy high inode space.

If it is not convenient to format the disk to increase the number of inode, follow these steps to clear files or directories with high inode usage.

  1. Log on to the server and run the following command to analyze the number of files in each secondary Directory of the root directory:
    for i in /*; do echo $i; find $i | wc -l; done
    The following command output is returned.

  2. Go to the directories with the highest inode usage layer by layer, run the preceding command, gradually locate the files or directories that consume too much space, and then clean them up accordingly.

Modify the number of inode

If you are not allowed to clean up files in the disk, or the inode usage is still high after cleaning, you need to increase the number of inode nodes by following these steps.

Note: to adjust inode, you need to reformat the disk. Make sure that the data has been properly backed up before performing the following operations.

  1. Run the following command to uninstall the system file:
    umount /home
  2. Run the following command to recreate the file system and specify the number of inode nodes:
    mkfs.ext3 /dev/xvdb -N 1638400
    Note: in this article, 1638400 inode nodes are specified. The actual on-site environment is subject to the actual situation.
  3. Run the following command to modify the fstab file:
    vim /etc/fstab
  4. Run the following command to view the modified Number of inode nodes:
    dumpe2fs -h /dev/xvdb | grep node
    The following command output is returned.

Analysis and deletion of zombie files

If neither the disk nor inode is faulty, check whether there are zombie files that have not been removed. These files have been deleted, but some programs are using them. As a result, these files are still in use and cannot be released disk space. Excessive files occupy a large amount of disk space. Follow these steps to view and delete zombie files:

  1. Remotely log on to the server.
  2. Run the following command to install lsof:
    yum install lsof -y
  3. Run the following command to view the usage of zombie files:
    lsof |grep delete | more
    The following command output is returned.

  4. Excessive zombie files occupy large disk space. You can release the handle using the following methods to clear the zombie file.
    1. Restart the server to verify the effect. Restart the server. The system exits the existing process and reloads it after the server starts up. This process releases the handle of the called distributed file.
    2. Run the lsof command to terminate or kill the process that occupies these files.
      Note: if the service on the server is running, the service may be affected. Perform this operation with caution.

Mount point overwrite

Cancel disk mounting and then check the space usage in the original Mount directory.

Application scope

  • ECS