Resilience and scaling

Being able to provide services that are resilient to errors and issues in the underlying infrastructure is one of the key reasons why we might want to use Kubernetes to deploy our containerized applications.

We are going to experiment with our Hello World deployment to discover how Kubernetes can deal with problems like these.

The first experiment is to see what happens when we deliberately remove the pod where our hello container is running.

To do this, we need to find the name of this pod, which we can do with the kubectl get command:

    $ kubectl get pods
    NAME                     READY     STATUS    RESTARTS   AGE
    hello-2473888519-jc6km   1/1       Running   0          7m

On our Minikube cluster, we currently only have one pod running from the one deployment that we have created so far. Once you start to deploy more applications, the output from commands such as kubectl get can get lengthier. We can use the -l flag to pass a label selector to filter down the results. In this case, we would use kubectl get pods -l run=hello to show just the pods where the run label is set to hello.

Then we can use the kubectl delete command to remove the resource. Deleting a pod also terminates the processes running inside of the constituent containers, effectively cleaning up the Docker environment on our node:

    $ kubectl delete pod/hello-2473888519-jc6km
    pod "hello-2473888519-jc6km" delete

If we then rerun the get pods command, you should notice that the pod we deleted has been replaced by a new one with a new name:

    $ kubectl get pod
    NAME                     READY     STATUS    RESTARTS   AGE
    hello-2473888519-1d69q   1/1       Running   0          8s

In Kubernetes, we can use replica sets (and deployments) to ensure that pod instances continue to run in our cluster despite unexpected events, be they a failed server, or a fat-fingered admin deleting our pod (as has happened in this case).

You should begin to understand as part of this exercise that a pod is an ephemeral entity. When it is deleted or the node it is running on fails, it is gone forever. Kubernetes ensures that the missing pod is replaced by another, created in its image from the same template. This means that any state that is stored on the local filesystem or in memory, the identity of the pod itself is also lost when a pod inevitably fails and is replaced.

This makes pods well-suited to some kinds of workload where it is not necessary for a state to be stored locally across runs, such as web applications and most batch jobs. If you are building new applications that you intend to deploy to Kubernetes, you will make them easier to manage by delegating the storage of state to an external store, such as a database or a service like Amazon S3.

We will explore features in Kubernetes that allow us to deploy applications that need to store local state and/or maintain a stable identity in Chapter 9, Storing State.

One problem you may have noticed when we were testing the abilities of Kubernetes to replace a pod that was removed is that, for a short time, our service became unavailable. For a simple example service running on a single node cluster such as this, perhaps this is not the end of the world. But we do really need a way for our applications to run in a way that minimizes even momentary downtime.

The answer is, of course, to ask Kubernetes to run multiple pin stances for our application, so even if one is lost, a second can take the slack:

    $ kubectl scale deployment/hello --replicas=2
    deployment "hello" scaled

If we now check the pods running, we can see a second hello pod has joined the party:

    $ kubectl get pods
    NAME                     READY     STATUS    RESTARTS   AGE
    hello-2473888519-10p63   1/1       Running   0          1m
    hello-2473888519-1d69q   1/1       Running   0          25m