How to Use journalctl - Linux Log Investigation Guide

How to Use journalctl - Linux Log Investigation Guide
Reading time: About 10 min Difficulty: Intermediate+

What you'll be able to do

  • Trace a failure by filtering logs down to a single service
  • Narrow logs by time, line count, and keyword
  • Confirm OOM kills and kernel errors from the kernel log

Prerequisites (read these first)

What You'll Learn

  • You can move past "where are the logs?" confusion in Ubuntu.
  • You can use journalctl to find the cause of a service failure quickly.
  • You can apply the practical patterns: "view recent only", "filter by time", "track while reproducing".
  • You can confirm OOM kills and kernel errors from the kernel log.

Quick Summary

For incident investigation, usually start with these:

  1. Check if service is dead: systemctl status <service>
  2. Recent logs: journalctl -u <service> -n 200
  3. Track while reproducing: journalctl -u <service> -f
  4. OS-level issues (OOM, etc.): journalctl -k | grep -i oom

Prerequisites

  • OS: Ubuntu
  • Target: Server beginners
  • systemd environment (most Ubuntu servers)
  • sudo access (some logs may not be visible otherwise)

Every command in this article only reads

journalctl displays logs. It never stops a service or rewrites a config, so it is safe to run on a production server. The only caution: without a line or time limit, the output gets huge. Filtering is covered below.

0. What is journalctl? (Minimal)

Conclusion: journalctl reads systemd's journald logs where startup and OOM events appear.

In Ubuntu, service logs may not be in files (/var/log/~) but collected in journald (journal). journalctl is the command to read this "aggregated log".

Settle the terms first.

Term One-line meaning Note
systemd The component that boots Linux and manages its services Also called the "init system"
journald The daemon that collects and stores logs for systemd Its full unit name is systemd-journald
journal The store journald keeps those logs in Also called the "journal log"
unit The general term for anything systemd manages The u in the -u option stands for this unit

Nginx/Apache have file logs too, but "service startup failure", "config error", "killed by OOM" often appear in journal.

1. First, Identify Service Name (Many Get Stuck Here)

Conclusion: Identify the exact service name first; systemctl status reveals it reliably.

Examples:

  • nginx → nginx
  • apache → apache2
  • ssh → ssh
  • php-fpm → php8.1-fpm, etc. (varies by environment)

Check status first to confirm the name:

$ sudo systemctl status nginx

The service name appears in this output.

2. View "Recent Only" (Most Used by Beginners)

Conclusion: Run journalctl -u <service> -n 200 for recent logs; 200 lines is a good start.

-u means "only logs from this service (unit)". -n 200 means "only the last 200 lines".

2-1. Last 200 Lines (Specific Service)

$ sudo journalctl -u nginx -n 200
Dec 15 13:02:11 web01 systemd[1]: Starting A high performance web server and a reverse proxy server...
Dec 15 13:02:11 web01 systemd[1]: Started A high performance web server and a reverse proxy server.
Dec 15 13:41:55 web01 nginx[1234]: 2025/12/15 13:41:55 [error] 1234#1234: *12 open() "/var/www/html/missing.html" failed (2: No such file or directory)

Each line reads: date and time, hostname, process name with its PID, then the message.

For Apache:

$ sudo journalctl -u apache2 -n 200

2-2. Shorter (Last 50 Lines)

$ sudo journalctl -u nginx -n 50

Start with 200 lines. Too few and "the key error isn't showing" happens often.

3. "Track While Reproducing" Is Most Powerful (-f)

Conclusion: Use journalctl -u <service> -f and reproduce in another tab to find causes fast.

This is the fastest path to the cause.

$ sudo journalctl -u nginx -f

-f (follow) keeps appending new log lines to your screen as they arrive. Press Ctrl+C to quit. You are only reading, so quitting has no effect on the service.

Reproduce with curl or browser in another tab → watch the log. This usually reveals the cause immediately.

4. Filter by Time (Makes Investigation Much Faster)

Conclusion: Filter by time with --since/--until once you know when the issue began.

If you know "when it broke", always filter by time.

4-1. Last Hour

$ sudo journalctl -u nginx --since "1 hour ago"

4-2. Today Only

$ sudo journalctl -u nginx --since "today"

4-3. Specific Time Range

$ sudo journalctl -u nginx --since "2025-12-15 13:00" --until "2025-12-15 14:00"

--since times are interpreted in the server's time zone. If the server runs on UTC, the result can be hours away from your local clock. When unsure, check the current zone with timedatectl (another read-only command).

5. Find Error Lines Only (Filter by Priority with -p)

Conclusion: Filter by priority with -p err first, then layer grep on top if needed.

5-1. Filter by priority (journalctl's own way)

journalctl records a priority for every log line. That is separate data from the message text, so -p can filter on it mechanically.

$ sudo journalctl -u nginx -p err --since "today"

The eight levels, most severe first, are emerg / alert / crit / err / warning / notice / info / debug. -p err shows only lines at err or above. It catches error lines that never spell out the word "error".

5-2. Filter by string (combined with grep)

When -p still leaves too much, narrow further by text.

$ sudo journalctl -u nginx --since "today" | grep -iE "error|fail|fatal|panic|denied|refused|timeout"

grep -i ignores letter case, and -E enables the | (or) syntax.

grep isn't perfect but great for initial investigation.

6. OS-Level Issues: OOM / Kernel Logs (Often Decisive)

Conclusion: App crashes may hide OOM or kernel issues; check them with journalctl -k.

When apps crash, processes die, 502s increase... OOM Killer (memory shortage) or kernel issues may be behind it.

The OOM Killer is the Linux mechanism that force-kills a process when memory runs out. "OOM" stands for Out Of Memory, and the app's own log may show nothing at all.

6-1. Kernel Logs (-k)

$ sudo journalctl -k -n 200

6-2. Find OOM Only

$ sudo journalctl -k | grep -i oom | tail -n 50
$ sudo journalctl -k | grep -i "killed process" | tail -n 50

7. Investigating "Startup Failure" (Common)

Conclusion: On startup failures, read systemctl status and journal together; -b scopes it.

When service won't start, check status and journal together:

7-1. status (Summary)

$ sudo systemctl status nginx

7-2. Recent Startup Logs (Specific Service)

$ sudo journalctl -u nginx -n 200

7-3. Current Boot Only (-b)

Logs "since current boot" only:

$ sudo journalctl -u nginx -b -n 200

-b means "since current boot".

8. Common Mistakes and Solutions

Conclusion: No logs, too many, or empty greps are typical; add sudo, filter, read raw logs.

8-1. "No Logs Appearing"

  • The service doesn't output to journald (file logs only)
  • Insufficient permissions to read

Solutions:

  • Add sudo
  • For Nginx/Apache, also check /var/log/nginx/ or /var/log/apache2/

Solutions:

  • Use --since for time filter
  • Use -n for line limit
  • Use -u for unit filter
  • Reproduce + -f (fastest)

8-3. "grep Finds Nothing" But Still Broken

Solutions:

  • grep conditions too narrow - first check raw logs with -n 200
  • Check error lines in status output

Things to Avoid

  • Spam restart and flush logs - Logs disappear and cause becomes invisible. First journalctl -u <service> -n 200.
  • Read everything without time filter - Wastes time. --since alone dramatically improves productivity.
  • Only check app logs, ignore OS - OOM and kernel issues don't appear in app logs. Always use journalctl -k.

9. When Logs Disappear After a Reboot

Conclusion: Some hosts keep no journal across reboots; journalctl -b -1 tells you which.

"The logs vanished after the reboot" often comes down to where journald stores them. journald writes either to /run/log/journal (in memory, cleared on reboot) or to /var/log/journal (on disk, kept across reboots).

Start by checking whether the previous boot is still readable.

$ sudo journalctl -b -1 -n 20

If the previous boot shows up, logs are persisted. If it reports that the boot cannot be found, that server keeps no journal across reboots.

Enabling persistence means editing /etc/systemd/journald.conf and restarting journald. That changes server behavior, so on production keep a copy of the original file and decide the disk cap (SystemMaxUse) at the same time. For an investigation, writing the current journalctl output to a file is the safer move.

$ sudo journalctl -u nginx -n 2000 > ~/nginx-journal-$(date +%Y%m%d).log

10. Investigation Checklist

Conclusion: State, scoping, time zone, kernel log, and preservation close the investigation.

  • [ ] You checked the current state with systemctl status <service>
  • [ ] You scoped with -u and limited the range with -n or --since
  • [ ] You confirmed the time zone with timedatectl and lined it up with the incident time
  • [ ] You checked journalctl -k for OOM kills and kernel errors
  • [ ] You wrote the logs you need out to a file

Copy-Paste Template

# 1) Check status first
sudo systemctl status <service>

# 2) Recent logs
sudo journalctl -u <service> -n 200

# 3) Track while reproducing (most powerful)
sudo journalctl -u <service> -f

# 4) Today only
sudo journalctl -u <service> --since "today"

# 5) OS-level (OOM, etc.)
sudo journalctl -k | grep -i oom | tail -n 50
sudo journalctl -k | tail -n 200

Next Reading

Share this article

Next steps

Next article → dmesg: Reading Kernel Ring Buffer Messages Practice with quiz → Verify your understanding Practice with lesson → Run commands in the terminal