Summary

We have come to the end of this chapter, and by now you should have a basic understanding of the Kafka messaging system. An important aspect of mastering any system is that you should understand the system end to end at a high level first. This will put you in a better position when you understand individual components of the system in detail. You can always establish the logical connection with end-to-end system understanding and understand why individual components are designed in a particular way. In this chapter, our goal was the same.

We started by discovering why Kafka was built in the first place. We have put forward problems in LinkedIn systems that led to the creation of Kafka. That section will give you a very clear understanding of the types of problem that Kafka can solve.

We further covered Kafka's logical and system architecture. Putting Kafka architecture in two viewpoints will help you with both a functional and technical understanding of Kafka. The logical viewpoint is more from the perspective of establishing data flows and seeing how different components depend on each other. The technical viewpoint will help you in technically designing producer/consumer applications and understanding the Kafka physical design. The physical viewpoint is more a system-wise view of the logical structure. The physical architecture covers producer Applications, consumer Applications, Kafka brokers (nodes), and Zookeeper.

In this chapter, we have touched on all components that we have illustrated in the Kafka architecture. We will cover all these components in depth in upcoming chapters. However, the important goal for you should be to understand the roles and responsibilities of each Kafka component. Every component in Kafka has some specific role to play, and, even if one of these is missing overall Kafka functionality cannot be achieved. The other key takeaways from this chapter should be understanding how the unit of parallelism and partitioning system works in Kafka. This is one of the key aspects in designing low'- latency systems with Kafka.

In the next chapter, we will delve into Kafka producers and how you should design a producer application. We will cover different producer APIs and some of the best practices associated with Kafka producers.