Fix Linux Disk Full and Inode Issues to Restore Service - Simple Application Server

If the error message No space left on device appears when you create a file or an application in your Linux simple application server, the disk space in the server is insufficient. If the disk usage is not as expected, you can identify the cause of the issue and resolve the issue by using the methods provided in this topic.

Note

If the disk usage is as expected, you can resolve the issue by upgrading your simple application server, attaching a data disk, or extending the existing data disk. For more information, see Upgrade a simple application server, Attach a data disk, or Extend the data disk.

Causes

The following causes may lead to insufficient disk space on a Linux instance:

The space usage of disk partitions reaches 100%.
The inode usage of disk partitions reaches 100%.
Leftover files (zombies) exist.
Note
File handles are still occupied by the deleted files. In this case, disk space that is consumed by deleted files cannot be released.
Mount points are overwritten.
Note
For example, the file system of a disk partition that manages a large number of files is mounted on a directory, which is referred to as a mount point. If the file system of another disk partition is mounted on the mount point, the mount point is overwritten by the new file system. When the applications on the system read and write the data of the original disk partition based on the mount point, the write or read operation fails and an error indicating that the disk space is insufficient is reported. When you run the du or df command to check the usage of the mount point, only the space usage of the new disk partition is queried.

Solutions

To resolve the issue, perform the following operations based on the cause of the issue:

1. The space usage of disk partitions reaches 100%.

To resolve this issue, you can delete files or directories that consume more disk space, resize disks, or create more disks. Perform the following operations:

Clear files or directories that occupy a large amount of space

Use the rescue feature to connect to the Linux simple application server. For more information, see Connect to a Linux server by using the rescue feature.
Run the following command to query the usage of disk space:
```
df -h
```
A command output similar to the following one is displayed. The command output indicates that the space usage of the /dev/xvda1 partition is 15%.
Run the following commands to enter the root directory and identify which directory consumes the largest amount of disk space:
```
cd /
du -sh *
```
A command output similar to the following one is displayed. The command output indicates that the /usr directory consumes the largest amount of disk space. Then, identify which file or directory consumes the largest amount of disk space in the /usr directory. Perform operations based on your actual scenarios.
Run the following commands in sequence to identify which directory consumes the largest amount of disk space in a hierarchical manner. For example, the preceding command output indicates that the /usr directory consumes the largest amount of disk space. Then, identify which file or directory consumes the largest amount of disk space in the /usr directory.
```
cd /usr
du -sh *
```
A command output similar to the following one is displayed. The command output indicates that the local directory consumes the largest amount of disk space in the /usr directory. Then, identify which file or directory consumes the largest amount of disk space in the local directory. Repeat the operation until the file or directory that consumes the largest amount of disk space is identified.
Delete files or directories that are no longer needed based on your business requirements.

Upgrade the simple application server, extend the existing data disk, or attach a data disk

If you cannot release more space by clearing files, you can upgrade the simple application server, extend the data disk, or attach a data disk to resolve the issue. For more information, see Upgrade a simple application server, Attach a data disk, or Extend the data disk.

2. The inode usage of disk partitions reaches 100%.

When the inode usage of disk partitions reaches 100%, directories or files cannot be created for applications even if the disk space is sufficient. In actual scenarios, you may be unable to take note that the inode usage already reached 100%. To resolve this issue, you can delete files or directories that use a large number of inodes or increase the number of inodes.

Note

In Linux, an inode contains the following information: the type, size, permissions, owner of a file, number of connections to the file, creation time and update time of the file, and pointers that point to data blocks. Modify inode configurations only when the inode usage reaches 100%.

Query the inode usage

Use the rescue feature to connect to the Linux simple application server. For more information, see Connect to a Linux server by using the rescue feature.
Run the following command to query the inode usage:
```
df -i
```
If the inode usage reaches 100% or is about to reach 100%, perform one of the following operations:
- Clear files or directories with high inode usage
- Increase the number of inodes

Clear files or directories with high inode usage

If you do not want to format disks to increase the number of inodes, perform the following operations to delete files or directories that use a large number of inodes:

Run the following command to query the number of files that exist in each subdirectory of the root directory:
```
for i in /*; do echo $i; find $i | wc -l; done
```
A command output similar to the following one is displayed. The command output indicates that the /usr directory has the largest number of files. Then, identify which directory in the /usr directory has the largest number of files. Inode usage increases with the number of files. Perform operations based on your actual scenarios.
Run the preceding command to identify which file or directory uses the largest number of inodes in a hierarchical manner. Then, delete the file or directory.

Increase the number of inodes

If files stored on disks cannot be deleted or if the inode usage remains high after files are deleted, back up disk data, format the disks, and then copy the data back to the disks to increase the number of inodes in the file systems.

Warning

To increase the number of inodes on disks, the disks must be formatted. In this case, the data stored on the disks is deleted. Before you format the disks, you must back up the data. You can copy files or create snapshots to back up data. For more information about how to create a snapshot, see the Create a snapshot section of the Manage snapshots topic.
To increase the number of inodes, you must unmount the file systems from mount points. However, this may interrupt your application services. We recommend that you unmount the file systems during an appropriate period of time.

Run the following command to unmount a file system from a mount point.
In this example, a file system is unmounted from the /home mount point. Modify the command based on your actual scenarios.
```
umount /home
```
Run the following command to create a file system and specify a larger number of inodes.
In this example, an ext3 file system is created for the /dev/xvdb partition and the number of inodes is set to 1,638,400. Modify the command based on your actual scenarios.
```
mkfs.ext3 /dev/xvdb -N 1638400
```
Note
In Linux, the number of inodes is subject to the disk capacity. In most cases, the number of inodes is calculated by using the following formula: Disk capacity (KB)/16 KB. For example, a 40 GB disk may have 2,621,440 inodes based on the preceding formula. The maximum number of inodes allowed for a disk is 2^32, which is approximately 4.3 billion. Specify an appropriate number of inodes based on the disk capacity.
Run the following command to mount the new file system to the mount point.
In this example, the new file system is mounted to the /home mount point based on the configurations in the /etc/fstab file. Modify the command based on your actual scenarios.
```
mount -a
```
Optional. Run the following command to view the number of inodes and check whether the number of inodes is increased:
```
dumpe2fs -h /dev/xvdb | grep node
```
A command output similar to the following one is displayed. The command output indicates that the number of inodes is increased. You can copy backup data back to the disk and restore the affected applications.

3. Leftover files (zombies) exist.

If no exceptions are detected for the space usage and inode usage of disk partitions, insufficient disk space on the instance may be caused by a large number of leftover files. If some files are deleted and displayed in the deleted state but are still used by processes, the disk space that is occupied by the files cannot be released and cannot be queried by running the df or du command. An excessively large number of leftover files occupy a large amount of disk space. Perform the following operations to query and delete leftover files:

Use the rescue feature to connect to the Linux simple application server. For more information, see Connect to a Linux server by using the rescue feature.
If lsof is not installed on the operating system, run one of the following commands to install lsof:
- Alibaba Cloud Linux and CentOS:
```
yum install -y lsof
```
- Debian and Ubuntu:
```
apt-get install -y lsof
```
Run the following command to query the space usage of leftover files:
```
lsof |grep delete | sort -k7 -rn | more
```
A command output similar to the following one is displayed. The sizes of files that are in the deleted state are displayed in the seventh column. Check whether the sum of the sizes is close to the unexpected usage of disk space. If the sum is close to the unexpected usage, it indicates that leftover files consume the disk space.
Use one of the following methods to release the file handles, clear the leftover files, and then release the disk space:
- Restart the instance
  After you restart the instance, the system terminates running processes and releases the handles of deleted files.
  Important
  Instance restarts may interrupt services. We recommend that you restart instances during an appropriate period of time.
- Run the kill command
  After you run the lsof command, the process IDs (PIDs) of running processes that correspond to the leftover files are often displayed in the second column. You can specify a PID in the kill command to terminate the corresponding process.
  1. Run the following command to list the PIDs of processes:
```
lsof |grep delete 
```
  2. Run the following command to terminate a process that corresponds to a leftover file based on your business requirements:
```
kill <PID>
```
    Important
    Services that run on instances may be affected when processes are terminated. Proceed with caution.

4. Mount points are overwritten.

If you troubleshoot the issue based on the preceding causes but the issue persists, the insufficient disk space may be caused by overwritten mount points. Use the following method to check whether overwritten mount points cause the issue:

In the example shown in the following figure, the space usage of the /dev/vda1 system disk whose size is 30 GB reaches 95%. After you run the du command, you can find that the /home directory consumes 24 GB of the disk space.

After you mount /dev/vdb1 on the /home directory, the space usage of the system disk /dev/vda1 is still 95%. However, in the root partition of the system disk, only the /usr directory consumes more than 1 GB of disk space and the /home directory consumes only 20 KB, but not 24 GB of disk space based on the previous check. You cannot identify which directory consumes the largest amount of disk space. The issue may be caused by overwritten mount points.

To resolve this issue, unmount the disk partition and check the space usage of the original mount point.

Warning

The unmount operation may interrupt your application services. We recommend that you unmount partitions during an appropriate period of time.