- Red Hat Enterprise Linux Troubleshooting Guide
- Benjamin Cane
- 5956字
- 2021-07-09 21:50:12
Data gathering
If we look back at Chapter 1, Troubleshooting Best Practices, the first step in the troubleshooting process is to understand the problem statement. In this section, we are going to explore how the problem was reported and will try to collect any data that we can to find the root cause of the issue.
For this example, we were notified of the issue via a phone call. This is actually lucky as we have an end user on the phone and can ask questions to get more information from him/her.
Before asking the person reporting the issue for more information, let's first take a look at what was already answered. All of a sudden, our blog is showing an installation page and not our posts!
At first, you may feel that this problem statement is vague; this is because it is vague. However, there is still quite a bit of useful information in this single sentence. If we dissect the reported issue, we can gain a better understanding of the problem.
- "Our blog is showing an installation page"
- "All of a sudden"
- "not our posts!"
From these three segments, we can assume the following:
- The blog is showing an unexpected page
- This blog was previously showing posts
- At some point, this changed and it seems that it was somewhat recently
While the above is a pretty good start for determining whether there is an issue and what it is related to, it does not give us enough to create a hypothesis yet.
Asking questions
In order to formulate a hypothesis, we will need more information. One method of getting this information is to ask the person reporting the issue. In order to get more information, we will ask the business user the following questions:
- When was the last time you saw the blog working?
Last night.
- What is the blog's address?
http://blog.example.com - Did you receive any other errors?
No.
While the above questions are not enough to identify the problem, they do give us a starting point of where to start looking.
Duplicating the issue
As previously stated in Chapter 1, Troubleshooting Best Practices one of the best methods of finding information is to duplicate the issue. In this case, it seems that we can duplicate the issue by simply going to the address provided.
In the previous screenshot, we can see that the blog is performing just as the user described. When we went to the provided URL, we were presented with a default WordPress installation screen.
Does this give us any clue about what the cause of the issue is? No, not really, not unless we have seen this issue before. While this may not tell us the cause of the issue, it does confirm that the issue that the user has reported is reproducible. This step has also told us the name of the software that we are troubleshooting: WordPress.
WordPress is one of the most popular open source blogging platforms. In this chapter, it is assumed that we have no experience managing WordPress and will need to find any information that we need around this web application through online sources.
Understanding the environment
Since we are the new systems administrator, at this point, we know very little about this environment, which means that we have little knowledge of how this blog is deployed. In fact, we do not even know which server it runs from.
One thing that we do know, however, is that all servers managed by our company have IPs within the 192.168.0.0/16 subnet. In order to determine whether this is an issue that we can resolve, we first need to determine whether the blog is on a server managed by our company. If this blog doesn't exist on a server managed by this company, our troubleshooting options may be limited.
One way to determine where the blog is hosted is to simply look up the IP address of the blog.example.com address.
There are many ways to look up the IP address of a DNS name; the command that we will discuss is the nslookup command. To use this command, simply execute nslookup followed by the DNS name to look up: blog.example.com for this example.
$ nslookup blog.example.com Server: 192.0.2.1 Address: 192.0.2.1#53 Non-authoritative answer: Name: blog.example.com Address: 192.168.33.11
In the preceding output, the result may be a bit confusing for those unfamiliar with nslookup.
Non-authoritative answer: Name: blog.example.com Address: 192.168.33.11
We know that the preceding information is the result of the nslookup query. This block is saying that the blog.example.com domain's address is 192.168.33.11. The first block of output from nslookup is simply telling us which DNS server was used to look up this information.
Server: 192.0.2.1 Address: 192.0.2.1#53
We can see from this block that the DNS server used was 192.0.2.1.
There are many commands that we could have used to look up the IP address of this domain. We could have used dig, host, or even ping. The reason that we chose the nslookup command is that for the most part, it is included with most operating systems. So, irrespective of whether you need to look up an IP address from a Windows, Mac, or Linux desktop, you can always use the nslookup command.
One caveat with the nslookup command, however, is that it specifically uses DNS to look up the address. It does not respect values in /etc/hosts or any other name service specified in /etc/nsswitch.conf. This is something that we will explore more in the later chapters; for now, we will assume that the IP address of 192.168.33.11 is the correct IP.
Since we are working with a Linux server, the most common way to manage that server is via Secure Shell (SSH). SSH is a secure network service that allows users to remotely access a server's shell. For this book, we are going to assume that you are already familiar with logging into a server via SSH. Whether you use the SSH command-line client or a desktop client like PuTTY, it is assumed that you are able to log into the server with SSH.
In this scenario, we use a laptop that has its own shell environment. To log into our server, we simply execute the ssh command from our terminal window.
$ ssh vagrant@blog.example.com vagrant@blog.example.com's password:
Once logged in, the first information-gathering command that we execute is the w command.
$ w 18:32:17 up 2 days, 12:05, 1 user, load average: 0.11, 0.08, 0.07 USER TTY LOGIN@ IDLE JCPU PCPU WHAT vagrant pts/1 00:53 2.00s 0.00s 0.08s sshd: vagrant [priv]
In Chapter 2, Troubleshooting Commands and Sources of Useful Information, we covered the w command and mentioned that it is the first command executed. We can see quite a bit of useful information in the output of the w command.
From this output, we can determine the following:
- Only 1 user is currently logged in (which is our login session)
- The server in question has been up for 2 days
- The load average is low, which suggests normal
Overall, at the first glance, the server seems to be performing normally. The fact that the issue started last night suggests that the issue did not start after the reboot 2 days ago. With the load average low, it is also safe at this point to assume that the issue is not related to the system load.
Since we have never logged into this server before, and are completely new to this environment, the first thing that we should do is find out what services are running on this server.
Since we know from the install page that the blog is a WordPress blog, we can search Google about the services that it requires. We can do this by using the search term "WordPress install requirements."
This search string returned with the following URL as the first result: https://wordpress.org/about/requirements/. This page contains the installation requirements for WordPress and lists the following:
- PHP 5.2.4
- MySQL 5.0 or higher
- Either Apache or Nginx web servers
From the fact that we can access the install page, we can assume that a web server and PHP are installed and somewhat working. However, it is always best to validate rather than assume.
Since WordPress recommends either the Apache or the Nginx web server, we first need to determine which is installed and, more importantly, identify which is in use for this WordPress application.
The following are a few ways to identify which web servers are installed and running:
- We could use
rpmto look at the packages installed - We could use
psto look at the processes running - We could simply go to a non-existent page via a browser and see whether the error page says which web server is running
- We can also go to
/var/logsand look around to see what log files exist or don't exist
All of these methods are valid and have their own benefits. For this example, we will use a 5th method (not mentioned earlier), which will answer two questions about the web server configuration on this server.
The first step of this method will be to determine which process is listening on port 80.
$ su - # netstat -nap | grep 80 tcp6 0 0 :::80 :::* LISTEN 952/httpd unix 3 [ ] STREAM CONNECTED 17280 1521/master
As discussed in Chapter 2, Troubleshooting Commands and Sources of Useful Information, the netstat command can be used to determine which ports are in use with the –na flags. If we simply add the –p (port) flag to netstat, we can also see which process is listening on each port.
Tip
In order to identify which processes are listening on each port, the netstat command must be executed with super user-level permissions. As such, we use the su command to switch to the root user before executing netstat.
Throughout this book, any command preceded with $ is run as an unprivileged user, while commands preceded with # are executed as the root user.
Port 80 is the default port for HTTP requests; as such, if we look back at the steps performed to duplicate the issue at hand, we can see that the address used was http://blog.example.com. Since this is an HTTP address and does not specify a different port, this means that the service that serves the WordPress installation page is listening on port 80.
From the output of the netstat command, we can see that process 952 is listening on port 80. The netstat output also shows that process 952 is running the httpd binary. On RHEL systems, this httpd binary is most often Apache.
We can validate whether this is the case with the ps command with the –elf flags discussed in Chapter 2, Troubleshooting Commands and Sources of Useful Information. We will also search the output of the ps command with the grep command, searching for the string "952":
$ ps -elf | grep 952 4 S root 952 1 0 80 0 - 115050 poll_s Jan11 ? 00:00:07 /usr/sbin/httpd -DFOREGROUND 5 S apache 5329 952 0 80 0 - 115050 inet_c 08:54 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND 5 S apache 5330 952 0 80 0 - 115050 inet_c 08:54 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND 5 S apache 5331 952 0 80 0 - 115050 inet_c 08:54 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND 5 S apache 5332 952 0 80 0 - 115050 inet_c 08:54 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND 5 S apache 5333 952 0 80 0 - 119196 inet_c 08:54 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
With the above output, we can see that process 952 and its child processes are running under the apache user. This confirms that the software in use is most likely Apache, but to be extra diligent, we can execute the httpd binary with the –version flag to print the version of the web server software.
$ httpd -version Server version: Apache/2.4.6 Server built: Jul 23 2014 14:48:00
The output of the httpd binary shows that it is in fact the Apache web server, which matches the WordPress requirements.
At this point, we have found out the following facts about the web server in use for this server:
It is possible to identify the same information by using other methods such as rpm. The good part of this method is that if the server has two web server services installed, we know which of these services is listening on port 80. This also tells us which service provides the WordPress install page.
A common WordPress implementation is to run the Apache, PHP, and MySQL services all on one server. Sometimes, however, the MySQL service will be run from another server or servers. To better understand the environment, we should check whether this environment runs MySQL locally or from another server.
To check this, we will once again use the ps command; this time, however, we will use grep to search for a process that matches the string "mysql":
$ ps -elf | grep mysql 4 S mysql 2045 1 0 80 0 - 28836 wait Jan12 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr 0 S mysql 2203 2045 0 80 0 - 226860 poll_s Jan12 ? 00:00:42 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log- error=/var/log/mariadb/mariadb.log --pid- file=/var/run/mariadb/mariadb.pid -- socket=/var/lib/mysql/mysql.sock
As you can see from the preceding output, there is in fact a MySQL process currently running. It is also important to note that the ps output shows that the mysqld process is using the following option: –log-error=/var/log/mariadb/mariadb.log.
This is important for two reasons: The first is that this is the location of the log file for the mysqld process, and the second is the fact that this log file is for MariaDB, which is different from MySQL.
We can confirm whether MySQL or MariaDB is installed by using the rpm and egrep commands.
$ rpm -qa | egrep "(maria|mysql)" php-mysql-5.4.16-23.el7_0.3.x86_64 mariadb-5.5.40-2.el7_0.x86_64 mariadb-server-5.5.40-2.el7_0.x86_64 mariadb-libs-5.5.40-2.el7_0.x86_64
The egrep command is similar to grep; however, it accepts search strings in the form of regular expressions. In the above command, we used egrep to search for either the string "mariadb" or the string "mysql." From the preceding output, we can see that this server does in fact have MariaDB installed but not MySQL.
With this information, we can make the assumption that the mysqld process that is running is in fact a MariaDB binary. We can verify this by using the rpm command with the –q (query) and –l (list all files) flags.
$ rpm -ql mariadb-server | grep "libexec/mysqld" /usr/libexec/mysqld
We can see from the rpm command's output that the running /usr/libexec/mysqld binary is deployed as part of the mariadb-server package. Showing that the running database process is in fact MariaDB and was installed via the mariadb-server package.
At this point, we have found the following facts about the database service running on this server:
- The database service is actually MariaDB
- MariaDB is running
- The log files for this service are at
/var/log/mariadb/
While MariaDB is a drop-in replacement for MySQL, the requirements for WordPress do not list it as the preferred database service. It is important to make note of this difference as it may determine the root cause of the reported issue.
Since we know that PHP is required for WordPress, we should also check whether it is installed. We can validate this by again using the rpm command.
$ rpm -qa | grep php php-mbstring-5.4.16-23.el7_0.3.x86_64 php-mysql-5.4.16-23.el7_0.3.x86_64 php-enchant-5.4.16-23.el7_0.3.x86_64 php-process-5.4.16-23.el7_0.3.x86_64 php-xml-5.4.16-23.el7_0.3.x86_64 php-simplepie-1.3.1-4.el7.noarch php-5.4.16-23.el7_0.3.x86_64 php-gd-5.4.16-23.el7_0.3.x86_64 php-common-5.4.16-23.el7_0.3.x86_64 php-pdo-5.4.16-23.el7_0.3.x86_64 php-PHPMailer-5.2.9-1.el7.noarch php-cli-5.4.16-23.el7_0.3.x86_64 php-IDNA_Convert-0.8.0-2.el7.noarch php-getid3-1.9.8-2.el7.noarch
PHP by itself is not designed to run as a service such as Apache or MySQL, but rather as a web server module. However, it is possible to use a service such as php-fpm as an application server. This allows PHP to run as a service and to be called by an upstream web server.
To check whether this server runs php-fpm or any other service that frontends PHP, we can again use the ps and grep commands.
$ ps -elf | grep php 0 S root 6342 5676 0 80 0 - 28160 pipe_w 17:53 pts/0 00:00:00 grep --color=auto php
By using the ps command, we do not see any specific PHP service; however, when going to the blog, we were able to see the install page. This suggests that PHP is configured to run via Apache directly. We can validate this by executing the httpd binary again with the –M (modules) flag.
$ httpd -M | grep php php5_module (shared)
The –M flag will tell the httpd binary to list all of the loaded modules. Included in this list is php5_module, which means that this installation of Apache is able to run PHP applications via php5_module.
At this point, we have identified the following from our data collection:
- The WordPress requirement of Apache is installed and running
- The WordPress requirement of MySQL appears to be met by MariaDB, which is installed and running
- The WordPress requirement of PHP is installed and appears to be working
- It appears that WordPress is deployed in a single-server setup rather than a multi-server setup
We can assume for now that these facts mean that the issue is not caused by a missing WordPress requirement.
By gathering all of these data points, we have not only learned more about the environment that we are troubleshooting but also eliminated several possible causes of this issue.
Looking for error messages
Now that the installed and configured services have been identified, we know where to start looking for errors or helpful messages. In the next stage of data collection, we are going to look through the various log files of these services to try and to identify any errors that may indicate the cause of this issue.
Since Apache calls PHP when web requests are made, the most likely log file to contain PHP-related errors is the Apache error log. The default log location for RHEL's httpd package is /var/log/httpd/. However, we don't know just yet whether the running httpd service is the RHEL packaged version.
Since we don't know the location of Apache's logs, we will need to find them. One way to find log files is to simply look in /var/log for any file or folder that matches the name of the service in question. This solution, however, is a bit too simple for our example.
To find the location of the httpd log files, we will use a method discussed in Chapter 2, Troubleshooting Commands and Sources of Useful Information and search through the service's configuration files. The /etc folder is the default folder for system configuration files. It is also the standard location for service configurations. Therefore, it is fairly safe to assume that the /etc/ folder will contain a configuration file or folder for the httpd service.
# cd /etc/httpd/ # ls -la total 20 drwxr-xr-x. 5 root root 86 Jan 7 23:29 . drwxr-xr-x. 79 root root 8192 Jan 13 16:10 .. drwxr-xr-x. 2 root root 35 Jan 7 23:29 conf drwxr-xr-x. 2 root root 4096 Jan 7 23:29 conf.d drwxr-xr-x. 2 root root 4096 Jan 7 23:29 conf.modules.d lrwxrwxrwx. 1 root root 19 Jan 7 23:29 logs -> ../../var/log/httpd lrwxrwxrwx. 1 root root 29 Jan 7 23:29 modules -> ../../usr/lib64/httpd/modules lrwxrwxrwx. 1 root root 10 Jan 7 23:29 run -> /run/httpd
In the preceding commands, we can see that we can switch to the /etc/httpd folder, which contains several configuration files. Since we don't know which configuration file contains the logging configuration, we could spend quite a bit of time reading through each configuration file.
To make this process faster, we can use the grep command to search through all of the files for the string "log." Since the /etc/httpd/ folder contains subfolders, we can simply add the –r (recursive) flag to cause the grep command to search through files contained in these subfolders.
# grep -r "log" /etc/httpd/* ./conf/httpd.conf:# with "/", the value of ServerRoot is prepended -- so 'log/access_log' ./conf/httpd.conf:# server as '/www/log/access_log', whereas '/log/access_log' will be ./conf/httpd.conf:# interpreted as '/log/access_log'. ./conf/httpd.conf:# container, that host's errors will be logged there and not here. ./conf/httpd.conf:ErrorLog "logs/error_log" ./conf/httpd.conf:# LogLevel: Control the number of messages logged to the error_log. ./conf/httpd.conf:<IfModule log_config_module> ./conf/httpd.conf: <IfModule logio_module> ./conf/httpd.conf: # define per-<VirtualHost> access log files, transactions will be ./conf/httpd.conf: # logged therein and *not* in this file. ./conf/httpd.conf: #CustomLog "logs/access_log" common ./conf/httpd.conf: # If you prefer a log file with access, agent, and referer information ./conf/httpd.conf: CustomLog "logs/access_log" combined ./conf.modules.d/00-base.conf:LoadModule log_config_module modules/mod_log_config.so ./conf.modules.d/00-base.conf:LoadModule logio_module modules/mod_logio.so ./conf.modules.d/00-base.conf:#LoadModule log_debug_module modules/mod_log_debug.so
While there is quite a bit of output from the preceding grep command, if we review the returned data, we can see that there are actually two log files defined for the httpd service: logs/access_log and logs/error_log.
./conf/httpd.conf:ErrorLog "logs/error_log" ./conf/httpd.conf: CustomLog "logs/access_log" combined
The defined logs use the relative path of logs/; this path is relative to the httpd services running folder. In this case, this means that the logs' folder is actually /etc/httpd/logs; however, this may not always be the case. To validate whether this is the case, we can simply perform a folder listing with the ls command in the /etc/httpd folder.
# ls -la /etc/httpd | grep logs lrwxrwxrwx. 1 root root 19 Jan 7 23:29 logs -> ../../var/log/httpd
From the ls command, we can see that /etc/httpd/logs exists; however, this is not a folder but a symbolic link to /var/log/httpd/. This means that the two log files, namely access_log and error_log, are actually located within the /var/log/httpd/ folder.
Now that we know where the log files are located, we can search these log files for any useful information. To do this, we will use the tail command.
The tail command is a useful command that can be used to read the last part of a file or files. By default, when tail is executed without any flags, the command will print the last 10 lines of the specified file.
For our troubleshooting, we want to not only see the last 10 lines of data but also watch the file for any new data being appended. To do this, we can use the –f (follow) flag, which tells tail to follow the specified file or files.
# tail -f logs/access_log logs/error_log ==> logs/access_log <== 192.168.33.1 - - [12/Jan/2015:04:39:08 +0000] "GET /wp-includes/js/wp-util.min.js?ver=4.1 HTTP/1.1" 200 981 "http://blog.example.com/wp-admin/install.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" "http://blog.example.com/wp-admin/install.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" 192.168.33.1 - - [12/Jan/2015:04:39:08 +0000] "GET /wp-admin/js/password-strength-meter.min.js?ver=4.1 HTTP/1.1" 200 737 "http://blog.example.com/wp-admin/install.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" ::1 - - [13/Jan/2015:16:08:33 +0000] "GET / HTTP/1.1" 302 - "-" "curl/7.29.0" 192.168.33.11 - - [13/Jan/2015:16:10:19 +0000] "GET / HTTP/1.1" 302 - "-" "curl/7.29.0" ==> logs/error_log <== [Sun Jan 11 06:01:03.679890 2015] [auth_digest:notice] [pid 952] AH01757: generating secret for digest authentication ... [Sun Jan 11 06:01:03.680719 2015] [lbmethod_heartbeat:notice] [pid 952] AH02282: No slotmem from mod_heartmonitor [Sun Jan 11 06:01:03.705469 2015] [mpm_prefork:notice] [pid 952] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations [Sun Jan 11 06:01:03.705486 2015] [core:notice] [pid 952] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
While there are no immediate PHP errors caused by the last 10 lines of each file, this does not necessarily mean that these files will not show the errors that we need. As this is a web-based application, we may need to load the application in order to trigger any errors.
We could simply open our browser and once again navigate to http://blog.example.com. However, for this example, we will utilize a very useful troubleshooting command: curl.
The curl command can be used as a client to access many different types of protocols, everything from FTP to SMTP. This command is particularly useful when troubleshooting a web application as it can be used as an HTTP client.
When troubleshooting a web application, you can use the curl command to make an HTTP, GET, or POST request to a specified URL, which when placed in the verbose mode with the –v (verbose) flag can produce quite a bit of interesting information.
$ curl -v http://blog.example.com * About to connect() to blog.example.com port 80 (#0) * Trying 192.168.33.11... * Connected to blog.example.com (192.168.33.11) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: blog.example.com > Accept: */* > < HTTP/1.1 302 Found < Date: Tue, 13 Jan 2015 21:10:51 GMT < Server: Apache/2.4.6 PHP/5.4.16 < X-Powered-By: PHP/5.4.16 < Expires: Wed, 11 Jan 1984 05:00:00 GMT < Cache-Control: no-cache, must-revalidate, max-age=0 < Pragma: no-cache < Location: http://blog.example.com/wp-admin/install.php < Content-Length: 0 < Content-Type: text/html; charset=UTF-8 < * Connection #0 to host blog.example.com left intact
The preceding output shows four key pieces of information that I want to highlight.
* Connected to blog.example.com (192.168.33.11) port 80 (#0)
The preceding line shows us that when we addressed the page called blog.example.com, we did in fact go to the server at 192.168.33.11. While we already identified that blog.example.com resolved to 192.168.33.11, this line confirms that the output from this command produces data from the expected system.
< HTTP/1.1 302 Found
The second key piece of information shows the HTTP status code that was provided by the web server.
In this case, the web server replied with a status code of 302, which is used to indicate a temporary redirect. When a browser requests a page and the web server replies with a 302 status code, the browser knows to redirect the end user to another page.
< Location: http://blog.example.com/wp-admin/install.php
The next page is determined by the Location HTTP header. This header, which is assigned by the web server, along with the HTTP status code of 302 will cause any browser to redirect the end user to the /wp-admin/install.php page.
This explains why we see an installation page when we navigate to blog.example.com as the web server simply responds with this 302 redirect.
< X-Powered-By: PHP/5.4.16
The fourth key piece of information is the HTTP header X-Powered-By; this is an HTTP header added by PHP. This header is added by PHP when the requested page is processed as by PHP, which means that our curl request was actually processed by PHP.
More importantly, we can see that the version of PHP (5.4.16) meets our minimum requirements as outlined by WordPress.
We can see when requesting a non-PHP page that no X-Powered-By header is added in the web server reply. We can do this by requesting an invalid URL.
# curl -v http://192.168.33.11/sdfas * About to connect() to 192.168.33.11 port 80 (#0) * Trying 192.168.33.11... * Connected to 192.168.33.11 (192.168.33.11) port 80 (#0) > GET /sdfas HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 192.168.33.11 > Accept: */* > < HTTP/1.1 404 Not Found < Date: Tue, 13 Jan 2015 21:18:57 GMT < Server: Apache/2.4.6 PHP/5.4.16 < Content-Length: 203 < Content-Type: text/html; charset=iso-8859-1
As we can see from the output obtained when requesting a non-PHP based page, the X-Powered-By header is not present. This indicates that the web server did not process this page as PHP.
The presence of the X-Powered-By header tells us that when we requested the blog.example.com page, it was processed by PHP. This also means that the HTTP status code of 302 was a response provided by WordPress. This information is important as it means that PHP is most likely processing pages without any issue, thereby eliminating PHP as a possible root cause of the reported issue, at least for now.
We can validate this further by reviewing any log entries generated from the abovementioned web requests.
When making the abovementioned requests with curl, we should have caused new log messages to be appended to the two httpd logs. Since we were using the tail command to continuously follow the log files, we can return to our terminal and review the new messages.
==> logs/access_log <== 192.168.33.11 - - [13/Jan/2015:23:22:17 +0000] "GET / HTTP/1.1" 302 - "-" "curl/7.29.0"
After our HTTP request to the blog URL, the only entry in either log was the preceding one. However, this is only an informational log message and not an error that would explain the issue. However, the informational log message is also a key data point. If there were an issue with the PHP code or processing, an error message similar to the following would have been generated.
[Tue Jan 13 23:24:31.339293 2015] [:error] [pid 5333] [client 192.168.33.11:52102] PHP Parse error: syntax error, unexpected 'endif' (T_ENDIF) in /var/www/html/wp-includes/functions.php on line 2574
The absence of a PHP error actually confirms that PHP is working as expected. This when combined with the curl results leads us to confidently assume that PHP is not the root cause.
While the httpd service logs may not have shown us an error that could answer why this issue is appearing, they have allowed us to eliminate a possible cause. While troubleshooting, you will often find yourself ruling out many possible causes before finding the exact cause of an issue. The troubleshooting steps mentioned earlier are exactly that, thereby eliminating possible causes.
Verifying the database
Earlier while checking what services were running, we found that the MariaDB service was running. We did not, however, validate that we can access the service or that the WordPress application can access this database service.
To validate that we can access the MariaDB service, we can simply use the mysql command.
# mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 28 Server version: 5.5.40-MariaDB MariaDB Server Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
The mysql command is actually a MariaDB client command. When run from the command line as the root user (as shown above), the mysql command by default will log into the MariaDB service as the MariaDB root user. While this is the default behavior, it is possible to configure the MariaDB service to disallow direct root login.
The abovementioned results imply that MariaDB allows direct root login, which shows that the MariaDB service itself is up and accepting connections. What they do not reveal is whether or not the WordPress application can access the database.
To determine this, we will need to log into the MariaDB service with the same username and password as the application.
In order to connect to the MariaDB service with the same credentials as WordPress, we need to obtain these credentials. We could ask the person who reported the issue for these details, but being a business user, they most likely would not know. Even if they worked with WordPress daily, in general, the database username and password are configured by one person and only used during installation.
This means that we must find this information for ourselves. One way to do this is to look through the configuration for WordPress as every web application that connects to a database has to obtain the login credentials from somewhere, and the most common way to do this is to store them within a configuration file.
An interesting challenge to this approach is the fact that this chapter assumes that we have little-to-no knowledge of WordPress. Finding where WordPress stores its database credentials will be a bit tricky; this is particularly true since we also do not know offhand where the WordPress application is installed either.
What we do know is that WordPress is a web application served by the httpd service. This means that the httpd service will have the installation path defined somewhere within its configuration files.
The default configuration for httpd is to serve a single domain from a default folder. The default folder can change from distribution to distribution, but in general, for RHEL systems, it is placed under /var/www/html.
It is possible to configure httpd to serve multiple domains; this is done via a Virtual Hosts configuration. At this point, we do not know whether this system is configured to host multiple domains or one single domain.
With the default single-domain configuration, any and all domains pointing to the server's IP would serve the same .html or .php files. With Virtual Hosts, you can configure Apache to serve specific .html or .php files depending on the domain that the request is being made to.
We can determine whether the httpd service is configured for Virtual Hosts or a single domain by executing a simple grep command.
# grep -r "DocumentRoot" /etc/httpd/ /etc/httpd/conf/httpd.conf:# DocumentRoot: The folder out of which you will serve your /etc/httpd/conf/httpd.conf:DocumentRoot "/var/www/html" /etc/httpd/conf/httpd.conf: # access content that does not live under the DocumentRoot.
Since the /etc/httpd folder has a multiple subfolder, we once again used the –r (recursive) flag for grep. The command searched the entire /etc/httpd folder structure for the DocumentRoot string.
DocumentRoot is the Apache configuration item that specifies the local folder that contains the .html or .php files for the specified domain. The DocumentRoot setting will be present multiple times for systems that are configured for multiple domains and only one time for single-domain configurations.
From the output above, we can see that on this server, DocumentRoot is only defined once and set to /var/www/html. As this is the default for RHEL systems, it is fairly safe to assume that the httpd service is configured in a single domain-based configuration.
To validate that this is the installation folder of WordPress, we can simply execute the ls command to list the files and folders within this path.
# ls -la /var/www/html/ total 156 drwxr-xr-x. 5 root root 4096 Jan 9 22:54 . drwxr-xr-x. 4 root root 31 Jan 7 23:29 .. -rw-r--r--. 1 root root 418 Jan 9 21:48 index.php -rw-r--r--. 1 root root 4951 Jan 9 21:48 wp-activate.php drwxr-xr-x. 9 root root 4096 Jan 9 21:48 wp-admin -rw-r--r--. 1 root root 271 Jan 9 21:48 wp-blog-header.php -rw-r--r--. 1 root root 5008 Jan 9 21:48 wp-comments-post.php -rw-r--r--. 1 root root 3159 Jan 9 22:01 wp-config.php -rw-r--r--. 1 root root 2726 Jan 9 21:48 wp-config-sample.php drwxr-xr-x. 6 root root 77 Jan 9 21:48 wp-content -rw-r--r--. 1 root root 2956 Jan 9 21:48 wp-cron.php drwxr-xr-x. 10 root root 4096 Jan 13 23:25 wp-includes -rw-r--r--. 1 root root 2380 Jan 9 21:48 wp-links-opml.php -rw-r--r--. 1 root root 2714 Jan 9 21:48 wp-load.php -rw-r--r--. 1 root root 33435 Jan 9 21:48 wp-login.php -rw-r--r--. 1 root root 8252 Jan 9 21:48 wp-mail.php -rw-r--r--. 1 root root 11115 Jan 9 21:48 wp-settings.php -rw-r--r--. 1 root root 25152 Jan 9 21:48 wp-signup.php -rw-r--r--. 1 root root 4035 Jan 9 21:48 wp-trackback.php -rw-r--r--. 1 root root 3032 Jan 9 21:48 xmlrpc.php
From the ls command's output, we can see that WordPress is in fact installed in /var/www/html/. We can conclude this on the basis of the large number of .php files along with the "wp-" naming scheme of these files. This will be confirmed, however, by the next steps.
Now that we have identified the installation folder, we simply need to find the database credentials within the WordPress application's configuration files. Unfortunately, we are not very familiar with WordPress and do not know which of these files hold the database credentials.
So, how are we going to find them? By googling it, of course.
As we covered in Chapter 1, Troubleshooting Best Practices, Google can be a system administrator's best friend. Since WordPress is a common open source application, it is very likely that there will be online help documentation that covers how to configure or at least recover the database password.
To get started, we will simply search WordPress database configuration via Google. While searching Google, we find that one of the first results is linked to the WordPress forum where a user asked where to find the database details in WordPress. (https://wordpress.org/support/topic/finding-the-database-settings-in-wordpress).
The first answer was to look through the wp-config.php file.
To obtain the database details, we can read the wp-config.php file with the less command. The less command is a simple command that allows users to read files via the command line. This is particularly useful for large files as it buffers the output rather than simply dumping all contents to the screen as in the case of the cat command.
# less /var/www/html/wp-config.php // ** MySQL settings - You can get this information from your web host ** // /** The name of the database for WordPress */ define('DB_NAME', 'wordpress'); /** MySQL database username */ define('DB_USER', 'wordpress'); /** MySQL database password */ define('DB_PASSWORD', 'password'); /** MySQL hostname */ define('DB_HOST', 'localhost');
By reading the configuration file, we can clearly see the database credentials, which are conveniently located towards the top of the file. The following is a list of the details that we could extract from this file:
NAME(wordpress) of the database that WordPress is trying to usedefine('DB_NAME', 'wordpress');HOST(localhost) that WordPress is attempting to connect todefine('DB_HOST', 'localhost');- The
USER(wordpress) database that WordPress is trying to authenticate withdefine('DB_USER', 'wordpress'); PASSWORD(password) that it is using for authenticationdefine('DB_PASSWORD', 'password');
With the above details, we can connect to the MariaDB services in the same way that the WordPress application does. This will be a critical step in our troubleshooting process.
Now that we have the database credentials, we can test the connectivity again with the mysql command. To connect to MariaDB with a specific username and password, we will need to use the –u (user) and –p (password) flags with the mysql command.
# mysql –uwordpress -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 30 Server version: 5.5.40-MariaDB MariaDB Server MariaDB [(none)]>
In the preceding command, we can see that we added the username after the –u flag but did not include the password after–p. Since we did not include the password, the mysql client simply asked for the password after we hit enter. While it is possible to include the password after–p, this is considered a bad practice from a security perspective. It is always better to let the mysql client ask for the password to reduce the chances of the password being compromised by those looking through the command history.
From the mysql client connection, we can see that by using the same credentials as WordPress, we were able to log into the MariaDB service. This is important as the inability to connect to the database service would impact the WordPress application and could have been a possible cause of the reported issue.
Since we can connect to the MariaDB service by using the WordPress credentials, we should next validate whether the database structure exists and is intact.
Tip
In this section, we will be executing Structured Query Language (SQL) statements from the MariaDB command-line interface. These statements are not shell commands but SQL queries.
SQL is the standard language for interacting with relational databases such as MySQL, MariaDB, Postgres, and Oracle. While SQL is not necessarily a language that every administrator needs to know, it is my advice that any systems administrator that supports a significant number of databases should at least know the basics of SQL.
This is particularly true if the environment you support does not have specific database administrators that manage the database and the database services.
The first item to validate is that the database itself is created and accessible. We can do this by using the show databases query.
MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | test | | wordpress | +--------------------+ 3 rows in set (0.00 sec)
We can see that the WordPress database is in fact listed in this output, meaning that it exists. To validate that the WordPress database is accessible, we will use the use SQL statement.
MariaDB [(none)]> use wordpress; Database changed
With the Database changed result, it seems that we have confirmed that the database itself is created and accessible. Now, what about the tables within this database? We can validate that the database tables have been created by using the show tables query.
MariaDB [wordpress]> show tables; +-----------------------+ | Tables_in_wordpress | +-----------------------+ | wp_commentmeta | | wp_comments | | wp_links | | wp_options | | wp_postmeta | | wp_posts | | wp_term_relationships | | wp_term_taxonomy | | wp_terms | | wp_usermeta | | wp_users | +-----------------------+ 11 rows in set (0.00 sec)
From the results, it appears that quite a few tables exist.
Since we are new to WordPress, it is possible that we may be missing a table and not even know it. As WordPress is documented online quite extensively, we are likely to find a list of tables by googling WordPress list of database tables, which returns a very useful database description from the WordPress documentation pages: (https://codex.wordpress.org/Database_Description)
After comparing the output of the show tables query and the Database Description page, we find that no tables are missing.
Since we know which tables exist, we should check whether these tables are accessible; we can do this by running a select query.
MariaDB [wordpress]> select count(*) from wp_users; ERROR 1017 (HY000): Can't find file: './wordpress/wp_users.frm' (errno: 13)
At long last, we have found an error!
The error that we have found, however, is quite interesting as it is not an error that you would typically see from an SQL query. In fact, this error seems to indicate that there is an issue with a file that contains the table data.
At this point, after validating the database, we have learnt the following:
- MariaDB is accessible by both the root user and the WordPress application
- The database being accessed is created and accessible by the WordPress user
- An error is shown when querying one of the database tables
With this information, we can move to the next step of the troubleshooting process by establishing a hypothesis.
- Java應用與實戰
- Visual Basic 6.0程序設計計算機組裝與維修
- MySQL數據庫應用與管理 第2版
- Scala Design Patterns
- Python測試開發入門與實踐
- Animate CC二維動畫設計與制作（微課版）
- Silverlight魔幻銀燈
- Learning Data Mining with R
- Learning Hunk
- Serverless computing in Azure with .NET
- Instant Nancy Web Development
- OpenGL Data Visualization Cookbook
- Java SE實踐教程
- Laravel Application Development Blueprints
- PyQt編程快速上手