Ubuntu journalctl Basics: Log Investigation for Faster Root Cause Analysis

journalctl Basics - Log Investigation
Reading time: ~10 min Level: Intermediate

What You'll Learn

  • Move past "where are the logs?" confusion in Ubuntu
  • Use journalctl to quickly find service failure causes
  • Master practical patterns: "view recent only", "filter by time", "track while reproducing"

Quick Summary

For incident investigation, usually start with these:

  1. Check if service is dead: systemctl status <service>
  2. Recent logs: journalctl -u <service> -n 200
  3. Track while reproducing: journalctl -u <service> -f
  4. OS-level issues (OOM, etc.): journalctl -k | grep -i oom

Table of Contents

  1. What is journalctl?
  2. First, Identify Service Name
  3. View "Recent Only"
  4. "Track While Reproducing" (-f)
  5. Filter by Time
  6. Find Error Lines Only
  7. OS-Level Issues: OOM / Kernel Logs
  8. Investigating Startup Failures
  9. Common Mistakes and Solutions

Prerequisites

  • OS: Ubuntu
  • Target: Server beginners
  • systemd environment (most Ubuntu servers)
  • sudo access (some logs may not be visible otherwise)

0. What is journalctl? (Minimal)

In Ubuntu, service logs may not be in files (/var/log/~) but collected in journald (journal).
journalctl is the command to read this "aggregated log".

Nginx/Apache have file logs too, but "service startup failure", "config error", "killed by OOM" often appear in journal.

1. First, Identify Service Name (Many Get Stuck Here)

Examples:

  • nginx → nginx
  • apache → apache2
  • ssh → ssh
  • php-fpm → php8.1-fpm, etc. (varies by environment)

Check status first to confirm the name:

$ sudo systemctl status nginx

The service name appears in this output.

2. View "Recent Only" (Most Used by Beginners)

2-1. Last 200 Lines (Specific Service)

$ sudo journalctl -u nginx -n 200

For Apache:

$ sudo journalctl -u apache2 -n 200

2-2. Shorter (Last 50 Lines)

$ sudo journalctl -u nginx -n 50

Start with 200 lines. Too few and "the key error isn't showing" happens often.

3. "Track While Reproducing" Is Most Powerful (-f)

This is the fastest path to the cause.

$ sudo journalctl -u nginx -f

Reproduce with curl or browser in another tab → watch the log.
This usually reveals the cause immediately.

4. Filter by Time (Makes Investigation Much Faster)

If you know "when it broke", always filter by time.

4-1. Last Hour

$ sudo journalctl -u nginx --since "1 hour ago"

4-2. Today Only

$ sudo journalctl -u nginx --since "today"

4-3. Specific Time Range

$ sudo journalctl -u nginx --since "2025-12-15 13:00" --until "2025-12-15 14:00"

5. Find Error Lines Only (High Priority)

When logs are many, first filter for "error-like strings":

$ sudo journalctl -u nginx --since "today" | grep -iE "error|fail|fatal|panic|denied|refused|timeout"

grep isn't perfect but great for initial investigation.

6. OS-Level Issues: OOM / Kernel Logs (Often Decisive)

When apps crash, processes die, 502s increase...
OOM Killer (memory shortage) or kernel issues may be behind it.

6-1. Kernel Logs (-k)

$ sudo journalctl -k -n 200

6-2. Find OOM Only

$ sudo journalctl -k | grep -i oom | tail -n 50
$ sudo journalctl -k | grep -i "killed process" | tail -n 50

7. Investigating "Startup Failure" (Common)

When service won't start, check status and journal together:

7-1. status (Summary)

$ sudo systemctl status nginx

7-2. Recent Startup Logs (Specific Service)

$ sudo journalctl -u nginx -n 200

7-3. Current Boot Only (-b)

Logs "since current boot" only:

$ sudo journalctl -u nginx -b -n 200

-b means "since current boot".

8. Common Mistakes and Solutions

8-1. "No Logs Appearing"

  • The service doesn't output to journald (file logs only)
  • Insufficient permissions to read

Solutions:

  • Add sudo
  • For Nginx/Apache, also check /var/log/nginx/ or /var/log/apache2/

8-2. "Too Many Logs to Search"

Solutions:

  • Use --since for time filter
  • Use -n for line limit
  • Use -u for unit filter
  • Reproduce + -f (fastest)

8-3. "grep Finds Nothing" But Still Broken

Solutions:

  • grep conditions too narrow - first check raw logs with -n 200
  • Check error lines in status output

Things to Avoid

  • Spam restart and flush logs - Logs disappear and cause becomes invisible. First journalctl -u <service> -n 200.
  • Read everything without time filter - Wastes time. --since alone dramatically improves productivity.
  • Only check app logs, ignore OS - OOM and kernel issues don't appear in app logs. Always use journalctl -k.

Copy-Paste Template

# 1) Check status first
sudo systemctl status <service>

# 2) Recent logs
sudo journalctl -u <service> -n 200

# 3) Track while reproducing (most powerful)
sudo journalctl -u <service> -f

# 4) Today only
sudo journalctl -u <service> --since "today"

# 5) OS-level (OOM, etc.)
sudo journalctl -k | grep -i oom | tail -n 50
sudo journalctl -k | tail -n 200

Related Articles

Test Environment

Commands in this article were tested on Ubuntu 24.04 LTS / bash 5.2.

Next Reading