- Building Data Streaming Applications with Apache Kafka
- Manish Kumar Chanchal Singh
- 429字
- 2022-07-12 10:38:16
Consumer configuration
Creating Kafka consumer also requires a few mandatory properties to be set. There are basically four properties:
- bootstrap.servers: This property is similar to what we defined in Chapter 3, Deep Dive into Kafka Producers, for producer configuration. It takes a list of Kafka brokers' IPs.
- key.deserializer: This is similar to what we specified in producer. The difference is that in producer, we specified the class that can serialize the key of the message. Serialize means converting a key to a ByteArray. In consumer, we specify the class that can deserialize the ByteArray to a specific key type. Remember that the serializer used in producer should match with the equivalent deserializer class here; otherwise, you may get a serialization exception.
- value.deserializer: This property is used to deserialize the message. We should make sure that the deserializer class should match with the serializer class used to produce the data; for example, if we have used StringSerializer to serialize the message in producer, we should use StringDeserializer to deserialize the message.
- group.id: This property is not mandatory for the creation of a property but recommended to use while creating. You have learned in the previous section about consumer groups and their importance in performance. Defining a consumer group while creating an application always helps in managing consumers and increasing performance if needed.
Let's see how we set and create this in the real programming world.
Java:
Properties consumerProperties = new Properties();
consumerProperties.put("bootstrap.servers", "10.200.99.197:6667");
consumerProperties.put("group.id", "Demo");
consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumerProperties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(consumerProperties);
Scala:
val consumerProperties: Properties = new Properties();
consumerProperties.put("bootstrap.servers", "10.200.99.197:6667")
consumerProperties.put("group.id", "consumerGroup1")
consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") consumerProperties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
val consumer: KafkaConsumer[String, String] = new KafkaConsumer[String, String](consumerProperties)
The preceding code contains three specific things:
- Properties object: This object is used to initialize consumer properties. Mandatory properties discussed earlier can be set as a key-value pair, where the key would be the property name and value would be the value for the key.
- Deserializer: This is also a mandatory property that tells which deserializer class is to be used to convert ByteArray to the required object. Class can be different for key and value, but it should align with the serializer class used in producer while publishing data to the topic. Any mismatch will lead to a serialization exception.
- KafkaConsumer: Once properties are set, we can create a consumer object by passing this property to the class. Properties tell the consumer object about brokers IP to connect, the group name that the consumer should be part of, the deserialization class to use, and offset strategy to be used for the commit.
推薦閱讀
- Docker進階與實戰
- Java Web基礎與實例教程(第2版·微課版)
- Dependency Injection in .NET Core 2.0
- 看透JavaScript:原理、方法與實踐
- Full-Stack Vue.js 2 and Laravel 5
- Microsoft System Center Orchestrator 2012 R2 Essentials
- 精通Linux(第2版)
- 編程數學
- NetBeans IDE 8 Cookbook
- Python從入門到精通
- 從零開始學Android開發
- Delphi開發典型模塊大全(修訂版)
- MyBatis 3源碼深度解析
- MySQL數據庫應用實戰教程(慕課版)
- 數據結構與算法詳解