How to Handle inode Exhaustion - When Disk Is Not Full But Writes Fail

2026-05-22 Reading time: About 10 min Difficulty: Intermediate

What You Will Solve Here

• Understand why writes fail with No space left on device while df -h reports free space
• Use df -i to find which filesystem ran out of inodes
• Locate the offending directory and reclaim inodes safely by cause (sessions, mail, Docker, logs)

What Is inode Exhaustion and Why Does It Happen?

inode exhaustion is the state where the filesystem's metadata table is full even though data blocks are still free, so creating any new file fails.

A filesystem has two ceilings:

• Data blocks: the bytes of file contents. Shown by df -h.
• Inodes: per-file metadata (permissions, owner, size, block pointers). Shown by df -i. One file consumes one inode.

ext4 uses static inode allocation: the inode count is fixed at mkfs time (default ratio is roughly one inode per 16 KiB). A workload that creates many small files exhausts inodes long before it fills the disk.

1. Check inode Usage with df -i

Start with df -i. It reports inode usage per mount point.

$ df -ih

Filesystem      Inodes IUsed IFree IUse% Mounted on
/dev/sda1         3.8M  3.8M   12K  100% /
/dev/sdb1         6.4M   12K  6.4M    1% /data
tmpfs             1.0M     5  1.0M    1% /run

The row with IUse% at 100% (or 99%) is the mount point in trouble. In the example above, / is full of inodes.

2. Find the Directory Consuming inodes

Drill down from the offending mount point to the directories holding the most files.

2-1. Count files quickly under a path

$ sudo find /var -xdev -printf . | wc -c

Always pass -xdev so find does not cross into other mount points. -printf . emits a single dot per file and wc -c counts bytes — much faster than wc -l on full paths.

2-2. Aggregate file counts by subdirectory

$ sudo find / -xdev -type f -printf '%h\n' 2>/dev/null \
    | cut -d/ -f1-4 \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -20

 432105 /var/lib/php
 187220 /var/spool/postfix
  54221 /var/log/journal
   8021 /var/lib/docker
   ...

cut -d/ -f1-4 groups by depth 3 to reveal the overall shape. When a suspicious path appears, rerun with -f1-6 to drill deeper.

2-3. Count entries in a single directory

$ ls -1U /var/lib/php/sessions | wc -l

ls -1U skips sorting so it stays fast even with millions of entries. wc -l counts lines, which equals the file count.

3. Fix by Cause

3-1. PHP sessions (/var/lib/php/sessions)

Sessions pile up when session.gc_probability is low or the distro's sessionclean cron stopped running.

# Check count and age first
$ sudo find /var/lib/php/sessions -xdev -type f | head
$ sudo find /var/lib/php/sessions -xdev -type f -mtime +7 | wc -l

# Print before deleting (dry run), then delete
$ sudo find /var/lib/php/sessions -xdev -type f -mtime +7 -print
$ sudo find /var/lib/php/sessions -xdev -type f -mtime +7 -delete

On Debian/Ubuntu, verify that /etc/cron.d/php runs sessionclean on schedule.

3-2. Postfix mail queue (/var/spool/postfix)

A failing outbound mail or bounce storm can balloon the queue.

$ sudo mailq | tail -1            # queue count
$ sudo postqueue -p | tail -1     # same
$ sudo postsuper -d ALL deferred  # delete only deferred mail
$ sudo postsuper -d ALL           # delete EVERYTHING (high impact)

3-3. systemd journal fragments

$ journalctl --disk-usage
$ sudo journalctl --vacuum-time=7d   # delete entries older than 7 days
$ sudo journalctl --vacuum-size=200M # cap total to 200 MB

Set SystemMaxFiles= in /etc/systemd/journald.conf to prevent recurrence.

3-4. Docker overlay2 layers

$ docker system df
$ docker system prune -a --volumes   # remove unused images and volumes (review impact first)

See Identifying Docker disk usage for the full breakdown.

3-5. Failed log rotation

$ sudo ls -lh /var/log | head
$ sudo logrotate -d /etc/logrotate.conf   # dry run
$ sudo logrotate -f /etc/logrotate.conf   # force run

4. Safety Before Deleting

4-1. Always dry-run first

# Print what would match
$ sudo find /tmp -xdev -type f -mtime +30 -print

# Delete only after reviewing the list
$ sudo find /tmp -xdev -type f -mtime +30 -delete

4-2. Watch for files held open after unlink

A file deleted while a process keeps it open is unlinked but still open — disk and inode are not released until the process closes the descriptor.

$ sudo lsof +L1 | head    # files with link count 0 still open

Restart the offending process to release both inode and space.

4-3. Avoid xargs argument overflow

# BAD: command line may overflow or partially execute
$ rm $(find /var/lib/php/sessions -type f)

# GOOD: use -print0 / xargs -0 for safety
$ sudo find /var/lib/php/sessions -xdev -type f -mtime +7 -print0 \
    | sudo xargs -0 -r rm --

See Safely deleting files with find for the full pattern.

5. Prevent Recurrence: Permanently Increasing inodes

5-1. Inspect the current inode ratio

$ sudo tune2fs -l /dev/sda1 | grep -E 'Inode count|Block count|Inode size'

5-2. ext4: reformat to grow the inode count

# Back up first. -i sets bytes-per-inode; smaller value = more inodes
$ sudo mkfs.ext4 -i 8192 /dev/sdb1     # roughly one inode per 8 KB
$ sudo mkfs.ext4 -N 10000000 /dev/sdb1 # specify total count

5-3. Move to a dynamic-inode filesystem (XFS / Btrfs)

If you can provision a new volume for a workload that creates many small files, XFS or Btrfs eliminates the inode ceiling problem because both allocate inodes on demand.

5-4. Add monitoring

Scrape node_filesystem_files and node_filesystem_files_free (Prometheus node_exporter) and alert when IUse% exceeds 80%. Catching it before 100% gives you time to delete safely instead of in panic mode.

6. What Not to Do