Mastering journalctl: Complete Guide to Systemd Logs on Linux

TIP

Bookmark this guide and run man journalctl alongside—each section maps to real-world tasks you perform during on-call incidents.

journalctl is the Swiss-army knife for systemd logs. Whether you are debugging a failed boot, tracing a flaky service, or auditing security events, mastering journalctl will dramatically cut your mean-time-to-resolution. This guide teaches you everything—from the most common flags to advanced filtering, persistent storage, and log rotation.

Table of Contents
1. Understanding systemd-journal Basics
2. Viewing Logs Chronologically
3. Real-Time Monitoring (Tail-like)
4. Filtering by Time
5. Filtering by Service / Unit
6. Priority-based Filtering
7. Grep-like Field Matching
8. Output Formats
9. Persistent vs Volatile Journals
10. Managing Disk Usage
11. Boot Troubleshooting with journalctl
12. Exporting & Importing Logs
13. Securing journalctl Access
14. Performance Tips
15. Cheatsheet Recap
16. Conclusion

1. Understanding systemd-journal Basics

systemd stores logs in a binary format under /var/log/journal/ (persistent) or /run/log/journal/ (volatile).
Each entry contains structured metadata: timestamp, PID, UID, SYSLOG_IDENTIFIER, MESSAGE, etc.
journalctl is the user-space client to query these binary journals and output human-readable text.

2. Viewing Logs Chronologically

# Entire journal (might need sudo)
journalctl

# Reverse order (newest first)
journalctl -r

# Recent boot only
journalctl -b   # current boot
journalctl -b -1  # previous boot

Fast Scroll Keys

Key	Action
`g`	jump to oldest
`G`	jump to newest
`/pattern`	search forward
`n` / `N`	next / previous match

3. Real-Time Monitoring (Tail-like)

# Follow the log like `tail -f`
journalctl -f

# Follow nginx service only
journalctl -u nginx -f

Combine with priority filter to watch errors only:

journalctl -u sshd -p err..alert -f

4. Filtering by Time

# Since yesterday 6 pm
journalctl --since "2025-07-29 18:00:00"

# Between two timestamps
journalctl --since "10:00" --until "10:15"

# Last 1000 lines
journalctl -n 1000

Relative times like --since "2h" also work.

5. Filtering by Service / Unit

journalctl -u docker.service
journalctl -u sshd.service -u nginx.service --since today

Unit names autocompleted via shell can speed up triage.

6. Priority-based Filtering

Level	Numeric	Usage
emerg	0	Kernel panic, system unusable
alert	1	Must be fixed immediately
crit	2	Critical conditions
err	3	Error conditions
warning	4	Warning conditions
notice	5	Normal but significant
info	6	Informational
debug	7	Debugging

# Show warnings and higher
journalctl -p warning

# Range example (err..crit)
journalctl -p 3..2

7. Grep-like Field Matching

# Messages containing string
journalctl | grep -i "oom-killer"

# Structured field filter (fast!)
journalctl _PID=1234
journalctl _COMM=sshd
journalctl _SYSTEMD_UNIT=cron.service _UID=0

List available fields with journalctl -o verbose -n 1.

8. Output Formats

journalctl -o short # default
journalctl -o json  # machine-readable
journalctl -o cat   # message only (no metadata)
journalctl -o short-iso # ISO timestamp

Pipe JSON into jq for ad-hoc dashboards.

9. Persistent vs Volatile Journals

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Check with:

stat -c %s /var/log/journal/*/*.journal   # file sizes

10. Managing Disk Usage

# View current stats
journalctl --disk-usage

# Set a maximum size (ex: 500M)
sudo nano /etc/systemd/journald.conf
SystemMaxUse=500M

# Vacuum old logs (retention)
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=1G

11. Boot Troubleshooting with journalctl

# Kernel ring buffer + journal
journalctl -k

# Failed systemd units since boot
systemctl --failed

# Logs leading up to the crash
journalctl -b -1 -e  # previous boot, end of file

12. Exporting & Importing Logs

# Export to text
journalctl -u nginx --since yesterday > nginx.log

# Export binary cursor-aware
journalctl --vacuum-time=30d --output=export > journal-export.bin

# Import on another machine
systemd-journal-remote --output=/tmp/remote.journal < journal-export.bin

13. Securing journalctl Access

Members of the systemd-journal group can read system logs without root.
Use sudo journalctl --verify to check for tampering; it validates SHA-256 seals.
Forward critical logs to SIEM via systemd-journal-remote and TLS.

14. Performance Tips

Tip	Benefit
Use field filters (`_PID=`) instead of grep	Faster, index-based searches
Limit output (`-n`, `--since`)	Reduces I/O & memory
Combine flags (`-u`, `-p`, `--since`)	Narrow scope for multi-unit debugging

15. Cheatsheet Recap

journalctl -b                # current boot
journalctl -u docker -f      # follow docker logs
journalctl -p err..alert -r  # recent errors across system
journalctl --since "2h"       # last 2 hours
journalctl _PID=1234 -o cat  # stdout from specific PID

16. Conclusion

With these commands in your arsenal, you can slice and dice systemd logs with surgical precision. Make journalctl part of your default troubleshooting workflow, script common queries for speed, and configure retention wisely to balance disk space and forensic needs. Happy debugging!

Table of Contents