Querying metrics

Targets are up and running, and Prometheus is scraping their data. We should generate some traffic that would let us see Prometheus query language in action.

We'll deploy go-demo stack. It contains a service with an API and a corresponding database. We'll use it as a demo service that will allow us to explore better some of the metrics we can use.

docker stack deploy \  
    -c stacks/go-demo.yml \  
    go-demo 

We should wait a few moments for the services from the go-demo stack to start running. Please execute docker stack ps go-demo to confirm that all the replicas are running.

Now that the demo service is running, we can explore some of the metrics we have at our disposal by opening graph page.

open "http://$(docker-machine ip swarm-1)/monitor/graph"

Please type haproxy_backend_connections_total in the Expression field, and press the Execute button. The result should be zero connections on the backend go-demo_main-be8080. Let's spice it up by creating a bit of traffic:

for ((n=0;n<200;n++)); do 
    curl "http://$(docker-machine ip swarm-1)/demo/hello" 
done 

We sent 200 requests to the go-demo service.

If we go back to the Prometheus UI and repeat the execution of the haproxy_backend_connections_total expression, the result should be different. In my case, there are 200 backend connections from go-demo_main-be8080.

We could display the data as a graph by clicking the Graph tab.

How about memory usage? We have the data through cadvisor so we might just as well use it.

Please type container_memory_usage_bytes{container_label_com_docker_swarm_service_name="go-demo_main"} in the Expression field and click the Execute button.

The result is memory usage limited to the Docker service go-demo_main. Depending on the view, you should see three values in Console or three lines in the Graph tab. They represent memory usage of the three replicas of the go-demo_main service.

Finally, let's explore one of the node-exporter metrics. We can, for example, display the amount of available memory from each of the nodes.

Please type sum by (instance) (node_memory_MemFree) in the Expression field and click the Execute button.

The result is a representation of free memory for each of the nodes of the cluster:

Now that we had a very brief overview of the ways we can query metrics, we should start using them.