Deep Dive into Kafka Producers

In previous chapters, you have learned about messaging systems and Kafka architecture. While it is a good start, we will now take a deeper look into Kafka producers. Kafka can be used as a message queue, message bus, or data storage system. Irrespective of how Kafka is used in your enterprise, you will need an application system that can write data to the Kafka cluster. Such a system is called a producer. As the name suggests, they are the source or producers of messages for Kafka topics. Kafka producers publish messages as per Kafka protocols defined by the makers of Kafka. This chapter is all about producers, their internal working, examples of writing producers using Java or Scala APIs, and some of the best practices of writing Kafka APIs. We will cover the following topics in this chapter:

• Internals of a Kafka producer
• The Kafka Producer API and its uses
• Partitions and their uses
• Additional configuration for producers
• Some common producer patterns
• An example of a producer
• Best practices to be followed for a Kafka producer