System Logging: syslog, journald, and logger

2026-05-30 Reading time: About 14 min Difficulty: Intermediate

What You Will Achieve

• Write rsyslog routing rules in the "facility.priority" format
• Distinguish the lists of facilities and priorities
• Use the key journalctl options (-u / -b / -f / -p / --since)
• Switch systemd-journald from volatile to persistent storage
• Record arbitrary messages to syslog with logger
• Explain how logrotate prevents logs from growing without bound

This is the core of LPIC-1 objective 108.2 "Manage the system's logs". It covers the two systems, rsyslog and systemd-journald, plus journalctl / logger / logrotate that operate on them.

Which Logging System Should You Use?

On most modern distributions, systemd-journald receives logs first and, when needed, forwards them to rsyslog so they are also written to traditional text logs such as /var/log/messages. This two-stage setup is common.

journald retains structured metadata (unit name, PID, boot ID, and so on) and lets you filter flexibly with journalctl. rsyslog is plain text and is strong for forwarding to remote hosts and long-term retention. The exam asks about both.

How Do You Write rsyslog Routing Rules?

rsyslog routing is written as a pair of "selector (facility.priority) plus action (destination)". You put it in /etc/rsyslog.conf or in files under /etc/rsyslog.d/.

The format looks like this:

mail.info                       /var/log/mail.log
authpriv.*                      /var/log/secure
*.emerg                         :omusrmsg:*
cron.*                          /var/log/cron

The left side is the selector and the right side is the destination. mail.info means "the mail facility at priority info and above". Specifying a single priority like info matches that severity and everything more severe, which is a frequent exam point. The * in authpriv.* means all priorities, and *.emerg means emerg of all facilities.

Facilities and Priorities

A selector combines a facility (the category that produced the message) and a priority (severity).

Main facility values:

Priorities (severity, listed from lowest):

*.info matches info and above (info / notice / warning / err / crit / alert / emerg). To limit to a single priority, add =, as in mail.=info.

Forwarding Logs to a Remote Host

rsyslog can forward logs to another host. On the sending side, write the selector and the forwarding destination in the configuration file.

*.*     @192.168.1.10:514
*.*     @@192.168.1.10:514

A single @ forwards over UDP, and a double @@ forwards over TCP. This is used to aggregate logs to a central log server. The receiving side must enable reception on the relevant port.

What Are the Key journalctl Options?

journalctl is the command that searches and displays the logs collected by systemd-journald. Run with no arguments, it shows all logs oldest first.

Representative filtering options:

journalctl -u sshd.service
journalctl -b
journalctl -f
journalctl -p err
journalctl --since "2026-05-30 09:00:00" --until "2026-05-30 12:00:00"

Example run:

journalctl -u sshd --since today -p warning

May 30 09:14:22 host sshd[1421]: Failed password for invalid user test from 203.0.113.5 port 55012 ssh2
May 30 10:02:51 host sshd[1588]: error: maximum authentication attempts exceeded for root

The -p priority accepts the same words as rsyslog (emerg through debug) or numbers (0 through 7). -p err collects err and above together.

How Do You Make journald Logs Persistent?

The storage location of systemd-journald is controlled by Storage= in /etc/systemd/journald.conf. By default (auto), it stores persistently in /var/log/journal/ if that directory exists, and stores volatilely in /run/log/journal/ (a tmpfs in memory) otherwise.

[Journal]
Storage=persistent

Steps to make logs persistent:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage

Archived and active journals take up 112.4M in the file system.

Setting Storage=persistent, or creating /var/log/journal/ and restarting journald, keeps logs across reboots. You can check usage with journalctl --disk-usage.

How to Use the logger Command

logger is a tool that sends messages to syslog from the command line or a script. It is widely used to record the execution log of shell scripts.

logger "backup script started"
logger -p local0.info -t backup "nightly backup finished"

Confirming the recorded result:

logger -p local0.info -t backup "nightly backup finished"
journalctl -t backup -n 1

May 30 02:00:03 host backup[2840]: nightly backup finished

The tag set with -t backup can be filtered with journalctl -t backup. Embedding logger in a script lets you trace later, via the journal, whether work launched from cron and the like succeeded.

How Does logrotate Prevent Logs from Growing?

logrotate is the mechanism that periodically rotates log files so they do not grow without bound. The main configuration is /etc/logrotate.conf, and per-service configurations go under /etc/logrotate.d/.

Example of /etc/logrotate.conf:

weekly
rotate 4
create
compress
include /etc/logrotate.d

weekly rotates weekly, rotate 4 keeps 4 generations, create makes an empty log after rotation, and compress gzip-compresses old logs. include /etc/logrotate.d loads the per-service configurations.

A per-service configuration (example /etc/logrotate.d/nginx):

/var/log/nginx/*.log {
    daily
    rotate 14
    missingok
    notifempty
    postrotate
        systemctl reload nginx > /dev/null 2>&1 || true
    endscript
}

daily rotates daily and rotate 14 keeps 14 generations. missingok does not error if the file is absent, and notifempty skips rotation if empty. postrotate through endscript notifies the process to reload after rotation.

Checking behavior (only judging, not actually rotating):

logrotate -d /etc/logrotate.conf

rotating pattern: /var/log/nginx/*.log  after 1 days (14 rotations)
considering log /var/log/nginx/access.log
  log does not need rotating (log has been already rotated)

-d (debug) is a dry run: it does not rotate and only prints the decisions. After changing the configuration, the standard practice is to confirm with -d before applying it in production. Adding -f (force) rotates even when the conditions are not met.

Where Are the Key Log Files?

The location of the text logs that rsyslog writes differs by distribution. The exam asks about the difference between Red Hat-based and Debian-based systems.

To investigate authentication failures, look at /var/log/secure on Red Hat-based systems and /var/log/auth.log on Debian-based systems. In an environment with only systemd-journald, these text files may not exist; in that case, obtain the equivalent information with journalctl.

Common Mistakes

Typical patterns where beginners stumble in log management.

• Mistake 2: Misreading mail.info as "info only" - A single priority in a selector includes "that severity and everything more severe". To limit to info only, use = as in mail.=info.
• Mistake 3: Confusing facilities and priorities - auth / cron / mail and the like are the source (facility); err / warning / debug and the like are the severity (priority). Confusing them keeps routing rules from working as intended.
• Mistake 4: Believing logrotate runs the moment you configure it - logrotate evaluates conditions when invoked from cron / a timer. To confirm immediately, use logrotate -d (judge only) or -f (force).
• Mistake 5: Looking only at text logs and missing the journal - In a systemd environment, many service logs exist only in the journal. Even if there is no file under /var/log/, check with journalctl -u <service>.

Troubleshooting

Symptom: journalctl -b -1 shows no previous-boot logs

Cause: journald is volatile, so logs were lost on reboot

Check:

cat /etc/systemd/journald.conf | grep Storage
ls /var/log/journal 2>/dev/null

Fix: Create /var/log/journal/ and make storage persistent with sudo systemctl restart systemd-journald. Or set Storage=persistent.

Symptom: You wrote an rsyslog rule but no log is produced

Cause: A misspelled facility/priority in the selector, or a missing configuration reload

Check:

sudo rsyslogd -N1
systemctl status rsyslog

Fix: Validate the syntax with rsyslogd -N1 and reload with sudo systemctl restart rsyslog. Also confirm the existence and permissions of the destination directory.

Symptom: Logs are filling up the disk

Cause: Too many retained generations in logrotate, or a large journald limit

Check:

journalctl --disk-usage
du -sh /var/log/*

Fix: Review the rotate generation count and compress in logrotate. journald can shrink past data with journalctl --vacuum-size=200M.

Completion Checklist

• [ ] Can read and write the rsyslog selector format (facility.priority)
• [ ] Can distinguish the lists of facilities and priorities
• [ ] Can use journalctl -u / -b / -f / -p / --since
• [ ] Can run the steps to make journald persistent
• [ ] Can record a tagged message with logger
• [ ] Can confirm rotation decisions with logrotate -d

Summary

System logging is the foundation of incident investigation and security auditing. Master rsyslog facilities and priorities, journald persistence, and capacity management with logrotate, and you will reliably score on 108.2.

Next Reading

• Job Scheduling: cron and systemd timers
• The Boot Process and systemd
• Command Line Basics