How to Investigate "No Space Left on Device" on Ubuntu

1. First, Check Which Area is Full (df)

$ df -h

Key Points to Watch

• Rows with high Use% (e.g., above 90%)
• Mounted on (where it's mounted)

Common Patterns

• / (root) is full
• /var is on a separate partition and full (logs and service data accumulate here)
• /var/lib/docker is bloated when using Docker

Also Check for inode Exhaustion (Too Many Files)

If you still get "no space" errors despite having available capacity, it might be inode exhaustion.

$ df -ih

If IUse% is high (near 100%), you likely have many small files.

2. Find What's Growing (du)

Once you know which mount point is full from df, use du to narrow down the cause.

Get Overview of Large Directories Under Root

(When root / is full)

$ sudo du -x -h -d 1 / | sort -h

• -x: Don't cross mount boundaries (easier to isolate the cause)
• -d 1: Depth of 1 level (start coarse)
• sort -h: Sort by size

If you see large directories (e.g., /var, /home), drill down into them next.

When /var is Large (Logs, Docker, DB Accumulate Here)

$ sudo du -x -h -d 1 /var | sort -h

Common Bloat Candidates

• /var/log (logs)
• /var/lib (Docker, DB data)
• /var/cache (cache)

3. Troubleshoot Log Bloat (/var/log and systemd journal)

Check /var/log Size

$ sudo du -h -d 1 /var/log | sort -h

Find large files:

$ sudo find /var/log -type f -printf "%s %p\n" | sort -n | tail -n 20

Common Culprits

• Nginx: /var/log/nginx/access.log error.log
• Apache: /var/log/apache2/access.log error.log
• System: /var/log/syslog auth.log

Check systemd Journal Size (journalctl)

On Ubuntu, the journal (binary logs) can become bloated.

$ sudo journalctl --disk-usage

4. Docker Environment Check (Entry Point for Log/Image Bloat)

Docker environments tend to consume space with logs, images, and volumes.

Check Overall Docker Usage

$ docker system df

• Identify which of Images / Containers / Local Volumes is large

5. Emergency Response (Safe Approach): Free Up Space First

Without stopping the root cause (continuous logging, persistent errors), the problem will recur.
However, in emergencies, you need to free space first to restore services.

Delete Old Journal to Limit Capacity (Relatively Safe)

Example to limit to 1GB max:

$ sudo journalctl --vacuum-size=1G

Day-based approach (delete logs older than 7 days):

$ sudo journalctl --vacuum-time=7d

When Logs Keep Growing Abnormally (Root Cause Entry)

If Nginx/Apache/WordPress keeps generating errors, logs will keep growing and recur.

• Nginx/Apache error logs (error.log)
• WordPress/PHP-FPM error logs
• Application exception logs

First identify what errors are recurring, then stop/fix the causing service.

6. Verification

After making changes, always verify improvement with df.

$ df -h
$ df -ih