This article helps us understand why system logs are vital, highlights key log files, introduces journalctl for systemd-based systems, demonstrates essential log viewing and searching tools, and touches upon log rotation concepts to keep your system's historical record manageable. Let's unlock the secrets hidden in your logs!

Why Logs Matter: Your System's Diary

System logs are like a detailed diary kept by your computer. They record important events, errors, warnings, and informational messages from the operating system, services, and applications. Monitoring logs is crucial for:

  • Troubleshooting problems: When something goes wrong, logs are often the first place to look for clues.
  • Security auditing: Logs can reveal unauthorized access attempts or suspicious activity.
  • Understanding system behavior: See how services start, stop, or interact.
  • Performance analysis: Identify recurring issues or bottlenecks.

The primary directory for logs on Linux systems is typically /var/log/.

Key Log Files to Know (Traditional Syslog)

While modern systems often use journald, it's good to be aware of these common traditional log files you might still encounter or find applications writing to:

  • /var/log/syslog or /var/log/messages: General system messages and activity (often the main log).
  • /var/log/auth.log or /var/log/secure: Authentication-related events (logins, sudo attempts, SSH activity).
  • /var/log/kern.log: Kernel messages.
  • /var/log/dmesg: Kernel ring buffer messages, often showing hardware and driver information from boot time.
  • /var/log/boot.log: System boot messages.
  • Application-specific logs: Many applications like web servers (e.g., /var/log/apache2/access.log, /var/log/nginx/error.log) or databases have their own log files within /var/log/ or their own directories.

The Modern Way: journalctl for systemd Systems

Most modern Linux distributions use systemd, which includes its own logging system called the journal. The journalctl command is your primary tool for querying and viewing these logs. The journal stores logs in a structured, binary format, offering more features than plain text logs.

View all logs (oldest first):

journalctl

View logs in reverse order (newest first, most common use):

journalctl -r

Follow new messages in real-time (like tail -f):

journalctl -f

Show kernel messages only:

journalctl -k

Show messages since a specific time (e.g., "yesterday", "1 hour ago", "2023-06-03 10:00:00"):

journalctl --since "1 hour ago"
journalctl --since yesterday

Show messages for a specific systemd unit (service):

journalctl -u sshd.service
journalctl -u nginx.service

Filter by priority (e.g., err, warning, info, debug):

journalctl -p err # Show errors and worse (crit, alert, emerg)
journalctl -p warning

Combine options: Show errors for nginx.service since yesterday:

journalctl -u nginx.service -p err --since yesterday

Classic Log Sleuthing: tail, grep, & less

Even with journalctl, these classic tools are indispensable for viewing and searching text-based log files (like those in /var/log/ or application-specific logs).

tail -f (Follow)

This is your go-to for watching a log file in real-time. New lines added to the file will appear on your screen.

tail -f /var/log/syslog
sudo tail -f /var/log/auth.log # Some logs require sudo

Press Ctrl+C to stop following.

tail -n 200 /var/log/syslog: Shows the last 200 lines.

less (Pager)

Allows you to view large files page by page, search within them, and navigate easily. It's much better than opening a huge log file in a text editor.

less /var/log/nginx/access.log

Inside less:

  • Spacebar or PageDown: Next page.
  • b or PageUp: Previous page.
  • /search_term: Search forward for search_term.
  • ?search_term: Search backward.
  • n: Next match.
  • N: Previous match.
  • G: Go to the end of the file.
  • g: Go to the beginning of the file.
  • q: Quit.

grep (Global Regular Expression Print)

Searches for patterns (text strings or regular expressions) in files or input. It's incredibly powerful for filtering log output.

Find lines containing "error" in a file:

grep "error" /var/log/syslog

Case-insensitive search (-i):

grep -i "failure" /var/log/auth.log

Show lines before (-B), after (-A), or around (-C) a match:

grep -C 2 "CRITICAL" /var/log/my_app.log # Shows 2 lines before and 2 after

Count matching lines (-c):

grep -c "login attempt" /var/log/auth.log

Combine with other commands using pipes (|): Watch for errors in real-time:

sudo tail -f /var/log/syslog | grep -i "error"

Search for "Failed password" events from journalctl:

journalctl -u sshd --since yesterday | grep "Failed password"

Keeping Logs Tidy: Log Rotation (logrotate)

Log files can grow very large over time, consuming disk space and becoming difficult to manage. Log rotation is the process of archiving old log files and starting new ones.

The logrotate utility is commonly used on Linux systems to automate this. It typically runs daily as a cron job. Its configuration is usually found in /etc/logrotate.conf and additional application-specific configurations in /etc/logrotate.d/.

Key Log Rotation Concepts:

  • Frequency: How often logs are rotated (e.g., daily, weekly, monthly).
  • Size Threshold: Rotate when a log file reaches a certain size.
  • Number of Rotations: How many old log files to keep (e.g., keep the last 7 rotated logs).
  • Compression: Old log files are often compressed (e.g., with gzip) to save space.
  • Actions: Scripts can be run before or after rotation (e.g., to signal an application to close and reopen its log file).

You usually don't need to manage logrotate manually unless you're configuring logging for a new application, but understanding that it's happening in the background is important. If you see files like syslog.1, syslog.2.gz, these are rotated logs.

By mastering log viewing and understanding how logs are managed, you gain invaluable insight into your system's operations, making troubleshooting and security analysis much more effective ! 🎉