官术网_书友最值得收藏!

Consumer configuration

Creating Kafka consumer also requires a few mandatory properties to be set. There are basically four properties:

  • bootstrap.servers: This property is similar to what we defined in Chapter 3, Deep Dive into Kafka Producers, for producer configuration. It takes a list of Kafka brokers' IPs.
  • key.deserializer: This is similar to what we specified in producer. The difference is that in producer, we specified the class that can serialize the key of the message. Serialize means converting a key to a ByteArray. In consumer, we specify the class that can deserialize the ByteArray to a specific key type. Remember that the serializer used in producer should match with the equivalent deserializer class here; otherwise, you may get a serialization exception.
  • value.deserializer: This property is used to deserialize the message. We should make sure that the deserializer class should match with the serializer class used to produce the data; for example, if we have used StringSerializer to serialize the message in producer, we should use StringDeserializer to deserialize the message.
  • group.id: This property is not mandatory for the creation of a property but recommended to use while creating. You have learned in the previous section about consumer groups and their importance in performance. Defining a consumer group while creating an application always helps in managing consumers and increasing performance if needed.

Let's see how we set and create this in the real programming world.

Java:

Properties consumerProperties = new Properties();
consumerProperties.put("bootstrap.servers", "10.200.99.197:6667");
consumerProperties.put("group.id", "Demo");
consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumerProperties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(consumerProperties);

Scala:

val consumerProperties: Properties = new Properties(); 
consumerProperties.put("bootstrap.servers", "10.200.99.197:6667")
consumerProperties.put("group.id", "consumerGroup1")
consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer") consumerProperties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
val consumer: KafkaConsumer[String, String] = new KafkaConsumer[String, String](consumerProperties)

The preceding code contains three specific things:

  • Properties object: This object is used to initialize consumer properties. Mandatory properties discussed earlier can be set as a key-value pair, where the key would be the property name and value would be the value for the key.
  • Deserializer: This is also a mandatory property that tells which deserializer class is to be used to convert ByteArray to the required object. Class can be different for key and value, but it should align with the serializer class used in producer while publishing data to the topic. Any mismatch will lead to a serialization exception.
  • KafkaConsumer: Once properties are set, we can create a consumer object by passing this property to the class. Properties tell the consumer object about brokers IP to connect, the group name that the consumer should be part of, the deserialization class to use, and offset strategy to be used for the commit.
主站蜘蛛池模板: 湘潭县| 广汉市| 崇礼县| 百色市| 防城港市| 山东省| 毕节市| 文成县| 唐山市| 于田县| 正镶白旗| 潼关县| 什邡市| 白水县| 屏南县| 石泉县| 乌拉特后旗| 清涧县| 襄汾县| 福建省| 太原市| 阳西县| 塘沽区| 从化市| 正安县| 巧家县| 离岛区| 凉城县| 安西县| 甘南县| 喀喇沁旗| 宁夏| 修水县| 新丰县| 偃师市| 青岛市| 金川县| 蓬安县| 宜昌市| 晋中市| 贵州省|