Deep Dive into Kafka Consumers

Every messaging system has two types of data flows. One flow pushes the data to the Kafka queues and the other flow reads the data from those queues. In the previous chapter, our focus was on the data flows that are pushing the data to Kafka queues using producer APIs. After reading the previous chapter, you should have sufficient knowledge about publishing data to Kafka queues using producer APIs in your application. In this chapter, our focus is on the second type of data flow--reading the data from Kafka queues.

Before we start with a deep dive into Kafka consumers, you should have a clear understanding of the fact that reading data from Kafka queues involves understanding many different concepts and they may differ from reading data from traditional queuing systems.

In this chapter, we will cover different concepts of Kafka consumers. Overall, this chapter covers how to consume messages from Kafka systems along with Kafka consumer APIs and their usage. It will walk you through some examples of using Kafka consumer APIs with Java and Scala programming languages and take a deep dive with you into consumer message flows along with some of the common patterns of consuming messages from Kafka topics.

We will cover the following topics in this chapter:

• Kafka consumer internals
• Kafka consumer APIs
• Java Kafka consumer example
• Scala Kafka consumer example
• Common message consuming patterns
• Best practices