Ubuntu Disk I/O Troubleshooting: iostat / vmstat Guide

Disk I/O Troubleshooting - iostat/vmstat
Reading time: ~10 min Level: Intermediate

What You'll Learn

  • How to determine if disk I/O is the bottleneck when the server is slow
  • How to use iostat / vmstat to distinguish CPU wait vs disk congestion
  • How to escape the "don't know what's happening" state

Quick Summary

When suspecting disk I/O issues:

  1. CPU idle but slow? → iostat -xz 1
  2. High I/O wait? → Check %iowait
  3. Write congestion? → Check await / %util
  4. Confirm disk is the cause? → vmstat 1

Table of Contents

  1. What is "Slow Disk I/O"?
  2. Installing iostat
  3. Using iostat -xz
  4. Key Metrics (%iowait / %util / await)
  5. Common Patterns
  6. Using vmstat
  7. Confirming Disk is the Culprit
  8. Common Causes

1. What is "Slow Disk I/O"?

"Slow disk" usually means one of these:

  • Application waiting for disk response
  • Log writes getting backed up
  • DB / Docker / backups consuming I/O
  • CPU is idle but stuck waiting on I/O

Looking at CPU% alone will always mislead you.

2. Installing iostat

$ sudo apt update
$ sudo apt install -y sysstat

3. Using iostat -xz

$ iostat -xz 1

Options:

  • -x: Extended stats (essential)
  • -z: Hide zero rows (cleaner output)
  • 1: Update every 1 second

4. Key Metrics (Core Knowledge)

4-1. %iowait (CPU side)

  • 10%+: Notable I/O wait
  • 20%+: Disk I/O is almost certainly the bottleneck

CPU wants to work but is stuck waiting on disk.

4-2. %util (Disk side)

  • 70% or less: Plenty of headroom
  • 80-90%: Getting congested
  • 100% constant: Completely saturated

4-3. await (Wait time)

  • Few ms: Normal
  • 10ms+: Slow
  • 50ms+: Noticeably slow
  • 100ms+: Critical

4-4. r/s w/s (Read/Write rate)

Tells you whether reads or writes are the problem. Log bloat and DB writes spike w/s.

5. Common Patterns

Pattern A: CPU low but slow

  • %iowait high
  • %util high

Classic disk I/O bottleneck

Pattern B: Both CPU and I/O high

  • %user / %system high
  • %iowait also high

Heavy processing + heavy disk writes combined

Pattern C: %util low but still slow

  • %util low
  • await high

Storage itself is slow (network storage, EBS, etc.)

6. Using vmstat

$ vmstat 1

Key columns:

  • b: Processes waiting on I/O (high = trouble)
  • wa: I/O wait (10%+ = warning)

If r is low but b is high, I/O is the cause, not CPU.

7. Confirming Disk is the Culprit

If ALL of these are true, disk is almost certainly the cause:

  • %iowait is high
  • %util is high
  • await is spiking
  • vmstat b column is increasing

8. Common Causes

  • Docker (logs, layers, overlayfs)
  • Databases (MySQL / PostgreSQL)
  • Log bloat (access.log, error.log)
  • Backups (rsync, tar)
  • Heavy cron jobs

Copy-Paste Template

# Disk details (most important)
iostat -xz 1

# Overall view
vmstat 1

# Disk list
lsblk
df -h

Related Articles

Test Environment

Commands in this article were tested on Ubuntu 24.04 LTS / bash 5.2.

Next Reading