Role of Zookeeper

We have already talked a lot about Zookeeper in the previous sections. Zookeeper plays a very important role in Kafka architecture and it is very important for you to understand how it records the Kafka cluster state. Therefore, we are dedicating a separate section to the role of Zookeeper in the Kafka cluster. Kafka cannot work without Zookeeper. Kafka uses Zookeeper for the following functions:

• Choosing a controller: The controller is one of the brokers responsible for partition management with respect to leader election, topic creation, partition creation, and replica management. When a node or server shuts down, Kafka controllers elect partition leaders from followers. Kafka uses Zookeeper's metadata information to elect a controller. Zookeeper ensures that a new controller is elected in case the current controller crashes.
• Brokers metadata: Zookeeper records the state of each of the brokers that are part of the Kafka cluster. It records all relevant metadata about each broker in a cluster. The producer/consumer interacts with Zookeeper to get the broker's state.
• Topic metadata: Zookeeper also records topic metadata such as the number of partitions, specific configuration parameters, and so on.
• Client quota information: With newer versions of Kafka, quota features have been introduced. Quotas enforce byte-rate thresholds on clients to read and write messages to a Kafka topic. All the information and states are maintained by Zookeeper.
• Kafka topic ACLs: Kafka has an in-built authorization module that is defined as Access Control Lists (ACLs). These ACLs determine user roles and what kind of read and write permissions each of these roles has on respective topics. Kafka uses Zookeeper to store all ACLs.

The preceding points summarize how Zookeeper is used in the Kafka cluster and why a Kafka cluster cannot run without Zookeeper. In upcoming chapters, you will understand Zookeeper concepts in more technical depth.