Best practices

Hopefully, at this juncture, you are very well aware of Kafka Producer APIs, their internal working, and common patterns of publishing messages to different Kafka topics. This section covers some of the best practices associated with Kafka producers. These best practices will help you in making some of the design decisions for the producer component.

Let's go through some of the most common best practices to design a good producer application:

• Data validation: One of the aspects that is usually forgotten while writing a producer system is to perform basic data validation tests on data that is to be written on the Kafka cluster. Some such examples could be conformity to schema, not null values for Key fields, and so on. By not doing data validation, you are risking breaking downstream consumer applications and affecting the load balancing of brokers as data may not be partitioned appropriately.
• Exception handling: It is the sole responsibility of producer programs to decide on program flows with respect to exceptions. While writing a producer application, you should define different exception classes and as per your business requirements, decide on the actions that need to be taken. Clearly defining exceptions not only helps you in debugging but also in proper risk mitigation. For example, if you are using Kafka for critical applications such as fraud detection, then you should capture relevant exceptions to send e-mail alerts to the OPS team for immediate resolution.
• Number of retries: In general, there are two types of errors that you get in your producer application. The first type are errors that producer can retry, such as network timeouts and leader not available. The second type are errors that need to be handled by producer programs as mentioned in the preceding section. Configuring the number of retries will help you in mitigating risks related to message losses due to Kafka cluster errors or network errors.
• Number of bootstrap URLs: You should always have more than one broker listed in your bootstrap broker configuration of your producer program. This helps producers to adjust to failures because if one of the brokers is not available, producers try to use all the listed brokers until it finds the one it can connect to. An ideal scenario is that you should list all your brokers in the Kafka cluster to accommodate maximum broker connection failures. However, in case of very large clusters, you can choose a lesser number that can significantly represent your cluster brokers. You should be aware that the number of retries can affect your end-to-end latency and cause duplicate messages in your Kafka queues.
• Avoid poor partitioning mechanism: Partitions are a unit of parallelism in Kafka. You should always choose an appropriate partitioning strategy to ensure that messages are distributed uniformly across all topic partitions. Poor partitioning strategy may lead to non-uniform message distribution and you would not be able to achieve the optimum parallelism out of your Kafka cluster. This is important in cases where you have chosen to use keys in your messages. In case you do not define keys, then producer will use the default round-robin mechanism to distribute your messages to partitions. If keys are available, then Kafka will hash the keys and based on the calculated hash code, it will assign the partitions. In a nutshell, you should choose your keys in a way that your message set uses all available partitions.
• Temporary persistence of messages: For highly reliable systems, you should persist messages that are passing through your producer applications. Persistence could be on disk or in some kind of database. Persistence helps you replay messages in case of application failure or in case the Kafka cluster is unavailable due to some maintenance. This again, should be decided based on enterprise application requirements. You can have message purging techniques built in your producer applications for messages that are written to the Kafka cluster. This is generally used in conjunction with the acknowledgement feature that is available with Kafka Producer APIs. You should purge messages only when Kafka sends a success acknowledgement for a message set.
• Avoid adding new partitions to existing topics: You should avoid adding partitions to existing topics when you are using key-based partitioning for message distribution. Adding new partitions would change the calculated hash code for each key as it takes the number of partitions as one of the inputs. You would end up having different partitions for the same key.