Disk Full but df Shows Space: Deleted-File Handles and inode Exhaustion

2026-06-05 Reading time: About 9 min Difficulty: Intermediate

What You'll Learn

• Why you get No space left on device even though df shows free space
• Why df reports a full disk but du totals far less (deleted-file occupancy)
• How to split the two directions of the gap with lsof and df -i, then reclaim space safely

Why does df disagree with actual free space?

Conclusion: df only reads the aggregate counters in the filesystem superblock, so a deleted-but-open inode or an exhausted inode table never shows up in the capacity percentage.

df does not walk directories; it reads the counters the filesystem keeps (free blocks, free inodes). du instead traverses real paths and sums their sizes. That implementation gap is the root of the discrepancy, and it shows up in three typical patterns.

How do I check if a deleted file is holding space?

Conclusion: lsof +L1 lists files that are still open but have a link count of 0 (deleted); a large SIZE on one of those rows is exactly what drives the gap between df and du.

Running rm does not free an inode while a process still has the file open, so its blocks are never reclaimed. The classic case: a log was deleted with rm, but the service keeps writing to it.

$ sudo lsof +L1

+L1 shows only open files whose link count is below 1 (i.e. 0, deleted). Look for rows where NLINK is 0 and SIZE is large.

You can also grep for the deleted marker:

$ sudo lsof -nP / 2>/dev/null | grep '(deleted)'

COMMAND   PID  USER   FD   TYPE DEVICE     SIZE/OFF NODE NAME
nginx    1234  root    5w  REG  259,1   8589934592  131 /var/log/nginx/access.log (deleted)

Here nginx still holds an 8 GB deleted log open. The result: df is full, but du /var/log looks small.

How do I release the held space?

Conclusion: the clean fix is to restart (or reload) the offending process. If you need space immediately, truncate /proc/PID/fd/N to zero bytes to reclaim it without a restart.

Option A: restart / reload the process (recommended)

Once the open fd is closed, the inode is freed and the blocks return instantly.

$ sudo systemctl restart nginx

For log files, many services reopen their files on reload (systemctl reload nginx) rather than a full restart.

Option B: truncate the fd when you cannot restart

If the service must stay up, truncate the still-open file via /proc/<PID>/fd/<FD>.

$ sudo truncate -s 0 /proc/1234/fd/5

: > /proc/1234/fd/5 does the same. The data is lost, but the process keeps running and the space is reclaimed immediately.

How do I check for inode exhaustion?

Conclusion: when df -i shows IUse% at 100%, writes fail even with free capacity; locate the directory holding many tiny files with find and trim it.

If df -h shows space but writes still fail, suspect inodes.

$ df -i

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      6553600 6553600     0  100% /

IUse% at 100% means inode exhaustion. Find where the many small files live:

$ sudo find / -xdev -type f 2>/dev/null | cut -d/ -f1-3 | sort | uniq -c | sort -n | tail

Common culprits are session files, mail queues, and cache fragments (/var/lib/php/sessions, /var/spool, large numbers of small files). For detailed cleanup, see Fixing inode Exhaustion.

Why does df show ~5% free but writes still fail?

Conclusion: ext4 reserves 5% of blocks for root by default, so non-root processes get ENOSPC even when df shows free space; check and adjust the reserve with tune2fs.

ext4 reserves 5% of the filesystem for root by default (to keep the system recoverable). Non-root apps hit ENOSPC even though df shows space.

$ sudo tune2fs -l /dev/sda1 | grep -i 'reserved block'

Reserved block count:     6553600

On a data-only partition you can lower the reserve to reclaim it (do not lower it too far on the root filesystem):

$ sudo tune2fs -m 1 /dev/sda1

-m 1 sets the reserve to 1%. Keep the default 5% on /.

How do I prevent it from recurring?

Conclusion: rotate logs with logrotate copytruncate or a correct reload hook so files are reopened, and monitor inodes; add df -i and lsof +L1 to your routine checks, not just df.

• Log bloat: configure logrotate with copytruncate, or signal the service correctly after rotation (never just rm the active log)
• inode monitoring: add df -i to monitoring. Watching only capacity (%) misses inode exhaustion
• Check before deleting: run lsof <file> before removing a large file to see who has it open