Tracking system resources with journald

Log files contain system messages and output from services, the kernel, and all kinds of running applications. They can be very useful in many situations, for instance, to troubleshoot system problems and monitor services or other system resources, or doing security forensics after a breach of security. In this recipe, you will learn the basics of how to work with logging services using journald.

Getting ready

To complete this recipe, you will need a working installation of the CentOS 7 operating system with root privileges and a console-based text editor of your choice. Also, setting the time and date correctly is very crucial for the whole logging concept, so please apply the Synchronizing the system clock with NTP and the chrony suite recipe from Chapter 2, Configuring the System before using this recipe. Also, a basic knowledge of systemd and units can be advantageous. This is covered in the Knowing and managing background services recipe in this chapter. Journalctl uses less navigation to show output; please read the Navigating text files with less recipe from Chapter 2, Configuring the System if you don't know how to work with it.

How to do it...

On CentOS 7, we have a choice between two logging mechanisms called rsyslog and the journald log system, which is a component of the new systemd system manager, for viewing and managing logging information. Here, we will show you how to work with the journalctl command, which is the controlling client for the journald daemon:

1. To begin, log in as root and type the following command to view the whole journal log:

   journalctl
   

2. Next, we want to show only the messages within a specific time frame (change the date accordingly):

   journalctl --since "2015-07-20 6:00:00" --until "2015-07-20 7:30:00"
   

3. Afterwards, we want to filter the log system by all messages from the sshd service:

   journalctl -u sshd.service --since "yesterday"
   

4. Now, we want to show only messages with type error:

   journalctl -p err -b
   

5. To get the most verbose version of journalctl, use the verbose option:

   journalctl -p err -b -o verbose
   

6. To get a current view on the log output, use the following command (this is not less navigation—use the key combination Ctrl+C to exit this view):

   journalctl -f
   

How it works...

In CentOS 7, we can use the new journald logging system, which is a part of the systemd system management. It is a centralized tool that will log just about everything on your system including all output from the early boot over kernel to services and all program messages. The main advantage over other logging mechanisms is that you don't have to configure logging for each of your services or other resources, because everything is already set up for all applications that are controlled and running through the centralized systemd system.

So, what have we learned from this experience?

We began our journey by running the journalctl command, which when applied without any parameters show us the complete journal log, which includes everything from starting your system and capturing the first boot log entries to the latest system messages in the order they appeared, appending new messages to the bottom (chronological order). If your system has been running for a while, it can contain hundreds of thousands of lines of logging data, and is very impractical to work with in this raw form.

This output is constantly captured by the journald daemon, but is not written to text files as other logging systems such as rsyslog do it. Instead, it uses a structured and indexed binary file, which stores a lot of additional meta information such as user Id, timestamp, and so on, and which makes it easy to transform into all kinds of different output formats. This can be very convenient if you want to further process journal information by another tool. As you cannot read binary files, you will need the client journalctl for it, which is used to query the journald database. Since it is almost impossible to parse through this sheer amount of data manually, we then take advantage of journalctl's rich filtering options. First, we used the --since and --until parameters to extract all log messages within a specific time frame. The syntax for specifying the time and date here is very flexible and understands phrases such as yesterday or now, but we stick with the simple date syntax, YYYY-MM-DD HH:MM:SS. Next, we used journalctl's -u parameter to filter log messages for a specific unit type. We used it to filter messages coming from the sshd daemon service. We added another filter using the --since parameter, which tightens the result of the -u unit filter even more, outputting only sshd service results that occurred yesterday. The next filter we applied was using the parameter string, -p err -b, which filters the log database by priority or log level. Every log message can have an associated priority that determines the importance of the message. To find out more about different log levels, refer to the manual using the command line man 3 syslog (if this manual is not available, install it by typing yum install man-pages). Our command will print out all log messages labeled as error or above, which includes: error, critical, alert, or emergency.

Next, we used the same command parameters but added -o verbose, which gives the most verbose output of logging information. Lastly we presented the -f parameter (for follow), which will give us a live view of the latest log messages and leaves this connection open, appending any new messages to the end of the output when they occur. This is often useful to see how the system reacts if you are currently testing out settings or starting/stopping services.

Summing up, one can say that, on CentOS 7, two logging systems do coexist: the older rsyslog and the newer journald, with the latter being your primary tool of choice for troubleshooting your system. But remember that on CentOS 7, journald is not a full replacement for rsyslog though. There are some rsyslog features that are missing in journald, and also there are lots of tools and scripts, such as log digesting tools or monitoring suites such as Nagios, that work exclusively with rsyslog.

System administrators often face a big challenge troubleshooting system errors or unexpected server behaviors. Often, it's not easy to find the single point of failure by searching through massive amounts of different log file texts while applying regular expression searches or Linux command line kung fu. Journald provides a very convenient alternative by providing a powerful and well-defined centralized querying system to get the log file analysis done quickly and efficiently!