Seeing what the container sees

When logs aren't really enough to figure things out, the command to use is docker exec in order to execute a command on the running container that can include access to a full-blown shell:

$ docker run -d \
             -p 8080:80 \
             nginx
06ebb46f64817329d360bb897bda824f932b9bcf380ed871709c2033af069118

$ docker exec 06ebb46f ls -la /etc/nginx/conf.d/
total 12
drwxr-xr-x 2 root root 4096 Jul 26 07:33 .
drwxr-xr-x 3 root root 4096 Jul 26 07:33 ..
-rw-r--r-- 1 root root 1093 Jul 11 13:06 default.conf

In this case, we used docker exec to run the ls command in the container, but as-is that is not really a powerful debugging tool. What if we try to get that full shell within the container and examine it that way?

$ docker exec -it \
              06ebb46f /bin/bash
root@06ebb46f6481:/# ls -la /etc/nginx/conf.d/
total 12
drwxr-xr-x 2 root root 4096 Jul 26 07:33 .
drwxr-xr-x 3 root root 4096 Jul 26 07:33 ..
-rw-r--r-- 1 root root 1093 Jul 11 13:06 default.conf
root@06ebb46f6481:/# exit
exit

$ # Back to host shell

This time, we used -it, which is shorthand for -i and -t flags that combine to set up the interactive Terminal needed for a full shell access and then we use /bin/bash to run Bash within the container. The shell within the container is a much more useful tool here, but we are at the mercy of the container itself in terms of installed tooling since many images trim out any unnecessary packages from the image--in this case, the NGINX container doesn't have ps, which is an extremely valuable utility for finding causes of problems. Since containers are isolated throwaway components generally, sometimes it might be fine to add your debugging tools to the container in order to find out what is causing problems (though we will cover a much better way of doing this with pid namespaces joining in later chapters):

$ docker exec -it 06ebb46f /bin/bash

root@06ebb46f6481:/# ps  # No ps on system
bash: ps: command not found

root@06ebb46f6481:/# apt-get update -q
Hit:1 http://security.debian.org stretch/updates InRelease
Get:3 http://nginx.org/packages/mainline/debian stretch InRelease [2854 B]
Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Hit:4 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Hit:5 http://cdn-fastly.deb.debian.org/debian stretch Release
Fetched 2854 B in 0s (2860 B/s)
Reading package lists...

root@06ebb46f6481:/# apt-get install -y procps
<snip>
The following NEW packages will be installed:
libgpm2 libncurses5 libprocps6 procps psmisc
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 558 kB of archives.
After this operation, 1785 kB of additional disk space will be used.
<snip>

root@06ebb46f6481:/# ps
PID TTY TIME CMD
31 ? 00:00:00 bash
595 ? 00:00:00 ps

root@06ebb46f6481:/#

As you can see, adding any debug tooling to the container from its upstream distribution is easy, but be aware that once you find your issue, you should start a new container and remove the old one to clean up the leftover junk since it is wasting space and a new container will start from the image that did not have your newly-installed debugging tools added (in our case procps).

Another thing to keep in mind is that sometimes, the images prevent the installation of additional packages, so for those cases we will need to wait until later chapters to see how we can use namespaces to work in such constrained settings.

Sometimes, the container is locked into a limited user shell, and because of it, you will be unable to access or modify other parts of the system of the container. In such configurations, you can add the -u 0 flag to run the docker exec command as root (user 0). You can also specify any other username or user ID instead, but generally if you need a secondary user to work with on a container, root is what you want.