Elastic Compute Service:Resolve full disk space issues on Linux instances

Scenario 1: Disk space is exhausted

1. Check disk usage.

   Run sudo df -h to view disk usage for each mount point. If the Use% value is 100%, the corresponding space is full.

2. Clean up unnecessary files or folders.

   Use sudo du -sh <folder name>/* to check the size of files and subfolders in a specified directory. If needed, navigate into the directory and repeat this command level by level to identify large items.

   For example, use sudo du -sh /mnt/* to check the disk space used by files and subfolders under the /mnt directory.

3. If space remains insufficient after cleanup, scale out your disk.

Scenario 2: Inode resources are exhausted

Each file consumes one inode. If your disk contains many small files, inodes can be exhausted even when disk space remains available, preventing new files from being created.

1. Check inode usage.

   Run sudo df -i. If the IUse% value reaches 100%, inode resources are exhausted.

2. Clean up unnecessary files or folders.

   Use sudo du -sh --inodes <folder name>/* to view the number of inodes used by files and subfolders in a specified directory. If needed, navigate into the directory and use this command recursively to locate high inode usage.

   For example, use sudo du -sh --inodes /mnt/* to check the number of inodes used by files and subdirectories in the /mnt directory.

3. If inode count remains insufficient after cleanup, scale out your disk.

Scenario 3: Deleted files still occupy space

Even after a file is deleted, the system does not release its disk space if a process is still using it (i.e., holds an open file handle). The space is only freed when the process terminates or explicitly closes the file.

1. Install the lsof tool.

   Files that are deleted but still hold space cannot be seen with df or du. Use the lsof tool to list them.

   Alibaba Cloud Linux, CentOS

   sudo yum install -y lsof

   Debian, Ubuntu

   sudo apt install -y lsof

2. View storage space held by deleted files.

   sudo lsof | grep delete | sort -k7 -rn | more

   The seventh column in the output shows file size in bytes. Sum these values to calculate total unreleased space.

   tail      619544                        root    3r   REG    253,1   50000000    136507 /home/test_file (deleted)
   aliyun-se 347980                        root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347997 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347996 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347995 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347994 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347983 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347982 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)
   aliyun-se 347980 347981 aliyun-se       root    7uW  REG    253,1          0    262160 /tmp/AliyunAssistClientSingleLock.lock (deleted)

3. Record the process name and PID.

   Run sudo lsof | grep delete and note the COMMAND and PID fields to identify the process name and process ID.

4. Restart or stop the relevant service.

   Run sudo ps -ef | grep <PID> to confirm the process purpose. After evaluating the impact, restart or stop the service.

   Important

   Restarting or stopping a service may affect your business. Evaluate carefully and perform this action during an appropriate maintenance window.

Scenario 4: Mount point is overlaid

When a non-empty directory is used as a mount point for another device, its existing contents become hidden. However, processes that already have the directory open can still write to the underlying space. This “hidden” space consumption cannot be observed with the df command and can unexpectedly exhaust disk space.

1. View information about duplicate folders.

   Run sudo lsblk and check the MOUNTPOINT column for repeated directory names.

   sudo lsblk

   NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   vda    253:0    0   40G  0 disk 
   ├─vda1 253:1    0    2M  0 part 
   ├─vda2 253:2    0  200M  0 part /boot/efi
   └─vda3 253:3    0 39.8G  0 part /
   vdb    253:16   0   40G  0 disk 
   └─vdb1 253:17   0   40G  0 part /mnt
   vdc    253:32   0   40G  0 disk 
   └─vdc1 253:33   0   40G  0 part /mnt

   In this example, both partitions vdb1 and vdc1 are mounted to /mnt, indicating a risk of mount point overlay.

2. Unmount the file system.

   Important

   Unmounting a file system may interrupt services that depend on the path. Assess the risk and choose an appropriate time to perform this operation.

   You can obtain the <duplicate mount directory> from the previous step.

   sudo umount <duplicate mount directory>

   In this example, /mnt is the duplicate mount directory. Running sudo umount /mnt unmounts the most recently mounted device, vdc1.

3. Identify the device name of the overlaid mount point.

   Run sudo df -h to locate the device name associated with the currently active mount point.

   sudo df -h

   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs        3.7G     0  3.7G   0% /dev
   tmpfs           3.7G     0  3.7G   0% /dev/shm
   tmpfs           3.7G  524K  3.7G   1% /run
   tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup
   /dev/vda3        40G  4.5G   33G  12% /
   /dev/vda2       200M  5.8M  194M   3% /boot/efi
   /dev/vdb1        40G   40G     0  100% /mnt
   tmpfs           747M     0  747M   0% /run/user/0

   In this example, the partition currently mounted to /mnt is vdb1. Therefore, the device name of the overlaid mount point is vdb1.

4. Resolve the full disk space issue.

   1. Clean up unnecessary files or folders in the overlaid space.

      In this example, clean up the /mnt directory mounted by vdb1.

   2. If space remains insufficient after cleanup, scale out your disk and mount it to a different empty directory.

      In this example, the target device to scale out is vdb1.

Scenario 5: Docker-related files consume significant space

Docker generates many intermediate images, stopped containers, and build caches during operation. Over time, these accumulate and consume disk space.

1. Check Docker disk space usage.

   Run sudo df -h. If the Use% for a Filesystem of type overlay reaches 100%, Docker storage is full.

2. Assess Docker resource usage.

   Run sudo docker system df and examine the Size and RECLAIMABLE fields to understand space consumption.

   sudo docker system df

   TYPE            TOTAL      ACTIVE     SIZE       RECLAIMABLE
   Images          21         9          13.94GB    10.66GB（76%）
   Containers      9          5          30.09MB    0B（0%）
   Local volumes   6          6          259.9MB    0B（0%）
   Build Cache     0          0          0B         0B

   In the example, Docker images occupy 13.94 GB, of which 10.66 GB is reclaimable, and we recommend that you prioritize cleaning up unused images.

3. Clean up unnecessary files.

   If Docker files cannot be cleaned, follow the steps in Scenario 1: Disk space is exhausted.

   • Remove all stopped containers: run sudo docker container prune.

   • Remove all dangling images (untagged images): run sudo docker image prune.

   • Remove unused build cache: run sudo docker builder prune.

Scenario 6: Inotify watches limit reached

When running commands like sudo tail -f, you might see the error tail: cannot watch '...': No space left on device. This does not mean disk space is full. Instead, the system has reached the maximum number of inotify watches used to monitor file and directory changes. You must increase this limit.

1. Check the current inotify watches limit.

   Run sudo cat /proc/sys/fs/inotify/max_user_watches to view the current limit for inotify watches.

2. Increase the inotify watches limit.

   Raising this limit increases memory usage. Evaluate carefully before changing. The <new limit> should generally not exceed 524288.

   sudo sh -c "echo fs.inotify.max_user_watches=<new limit> >> /etc/sysctl.conf"

3. Load the new configuration.

   Run sudo sysctl --system to apply the updated configuration.

4. Verify the result.

   Run sudo cat /proc/sys/fs/inotify/max_user_watches again to confirm the inotify watches limit has been updated as expected.