Borg

This is where the Borg project comes in. Google uses containers a lot, and when I say a lot I mean a lot. In May 2014, Joe Beda from Google gave a presentation at Gluecon entitled Containers At Scale. There were a few takeaway quotes from the presentation such as:

"Everything at Google runs in a container."

And the one that gets talked about the most is:

"We start over 2 billion containers per week."

This works out at around 3,000 per second and, during the talk, it was mentioned that the number didn't include any long-running containers.

While Joe went into some detail about how Google was using containers at that time, he did not mention anything directly about the Borg project; instead, it was simply referred to as a cluster scheduler.

The final takeaway from the presentation was the slide entitled Declarative Over Imperative, which introduced the following concepts:

• Imperative: Start this container on that server 
• Declarative: Run 100 copies of this container with a target of <= 2 tasks down at any time

This concept explains how Google was able to launch those 2 billion containers per week without having to really manage well over 2 billion containers.

It wasn't until Google published a paper entitled Large-scale cluster management at Google with Borg in 2015 that we really got an insight into the practices and design decisions that went into the cluster scheduler mentioned by Joe Beda the previous year.

The paper talks about how Google's internal tooling, called Borg, runs thousands of jobs, which go to make up pretty much all of Google's applications across clusters made up of tens of thousands of machines.

It then goes on to reveal that customer-facing services such as Google Mail, Google Docs, and Google Search are all served from Borg-managed clusters as well as their own internal tools. It details the job specification language that users can use to declare their desired state, making it easy for users to deploy their applications without having to worry about all of the steps needed to deploy their application in a highly available configuration across Google's infrastructure.

I would recommend reading through the paper as it gives an excellent overview of how Google approaches its own container services.

Also, in case you are wondering, Borg was named after the alien race from the Star Trek: The Next Generation TV show.