Message producers

In Kafka, the producer is responsible for sending data to the partition of the topic for which it is producing data.

The hash value is generally calculated by the message key that we provide when writing the message to a Kafka topic. The message with a null key will be distributed in a round-robin fashion across partitions to ensure even distribution of messages. In Kafka, each partition has a leader and each read write request goes through the leader only. So a request to write messages to a partition of a topic will go through the leader broker. The producer waits for an acknowledgement of messages depending on the setting. Generally, it waits until the replication for a particular message is successfully acknowledged.

However, you can set the configuration for acknowledgement to 1, which assumes that if a message is committed by the leader, it will be available to read and the Kafka producer can produce the next messages. This setting is dangerous because, if brokers fail before other replicas commit the message, the message will be lost. This leads to less durability but high throughput.

However, it's better to compromise on throughput if your consumer system does not want to lose a single message as part of the application. We will talk in detail about the producer in the next chapter.