What do I do if the df and du command outputs are inconsistent? - Alibaba Cloud Linux

The df and du commands both report disk space usage but gather data differently. When their outputs disagree by gigabytes or more, a specific system condition is usually the cause.

Symptoms

df reports a file system at 90%+ usage, but du on the same mount point shows far less space consumed.
After deleting large files, df still shows the disk as full.
du -sh /mountpoint returns a total that does not match the "Used" column in df -h.

How df and du differ

The two commands use different system calls and report different things.

Aspect	`df`	`du`
Scope	File system level	Files and directories
System call	`statfs()` (reads superblock)	`stat()` (reads each file)
Default unit	1-KB blocks	KB
Traversal	None (reads superblock metadata)	Depth-first traversal of directory tree
Deleted-but-open files	Counted (space still allocated)	Not counted (file no longer in directory tree)
Block size option	`-B` or `--block-size=SIZE`	`-B` or `--block-size=SIZE`

Under normal conditions, both commands return similar totals. A large discrepancy points to one of the issues below.

Quick diagnostic triage

Run these commands to identify the cause:

Step	Command	If output is non-empty, go to
1	`sudo lsof +L1`	Issue 1: Deleted files still held open
2	`mount \| grep <mount_point>` (look for nested mounts)	Issue 2: Nested mount points

Issue 1: Deleted files still held open

Symptom

df reports the file system as nearly full. du reports significantly less usage on the same file system.

Cause

When a file is deleted (unlinked), the file system does not reclaim its space until every process holding an open file descriptor closes it. The file disappears from directory listings, so du no longer counts it. The file system still tracks the allocated blocks, so df still reports the space as used.

Diagnose

List all deleted files still held open by processes:

sudo lsof | grep deleted

Each line shows the process name, PID, and the size of the deleted file.

To calculate the total space consumed by all deleted-but-open files:

sudo lsof -Fn -Fs | grep -B1 -i deleted | grep ^s | cut -c 2- | awk '{s+=$1} END {print s}'

The output is in bytes. Divide by 1073741824 to convert to GB.

Resolve

Option A: Stop the process

If the process can be safely stopped, terminate it to release the file descriptor:

sudo kill -9 <PID>

Replace <PID> with the process ID from the lsof output.

After the process stops, run df -h to confirm the file system has reclaimed the space.

Option B: Truncate the file without stopping the process

In production environments where stopping the process is not acceptable, truncate the deleted file through the /proc file system:

Identify the PID and file descriptor number from the lsof output.
Truncate the file:

sudo sh -c 'echo > /proc/<PID>/fd/<FD_NUMBER>'

Replace <PID> with the process ID and <FD_NUMBER> with the file descriptor number.

Warning

The application holding the file may not handle truncation gracefully and could exhibit undefined behavior. Use this method only when stopping the process is not an option, and monitor the application afterward.

Issue 2: Nested mount points

Symptom

du reports a different total for a mount point than df reports for the same file system.

Cause

When a second file system is mounted to a subdirectory under an existing mount point, du traverses into the nested mount and counts files from both file systems. df reports only the space used by the original file system, so du returns a combined total while df returns usage of a single file system.

Example

Suppose /dev/vdb is mounted at /mnt/vdb with 30 GB used, and a 1-GB file exists in /mnt/vdb/tmp. A separate device /dev/vdc is then mounted at /mnt/vdb/tmp with 5.1 GB used.

Check the mount configuration:

sudo mount | grep /mnt/vdb

Output:

/dev/vdb         40G   30G   11G  75% /mnt/vdb
/dev/vdc         40G  5.1G   33G  14% /mnt/vdb/tmp

Run du on the parent mount point:

du -hs /mnt/vdb

Output:

34G     /mnt/vdb

du reports 34 GB because it traverses into /mnt/vdb/tmp and adds the 5.1 GB from /dev/vdc to the 30 GB from /dev/vdb. Meanwhile, df shows only 30 GB used for /dev/vdb.

Diagnostic tip: Use the -x flag with du to restrict it to a single file system and prevent it from crossing mount boundaries:

du -xhs /mnt/vdb

If the -x output matches df, nested mounts are the cause of the discrepancy.

Resolve

Unmount the nested file system to align df and du on the same file system:

sudo umount /mnt/vdb/tmp

If the nested mount is intentional and must remain, use du -x to exclude it from the total, or query each file system individually with df.