官术网_书友最值得收藏!

Additional producer configuration

There are other optional configuration properties available for Kafka producer that can play an important role in performance, memory, reliability, and so on:

  • buffer.memory: This is the amount of memory that producer can use to buffer a message that is waiting to be sent to the Kafka server. In simple terms, it is the total memory that is available to the Java producer to collect unsent messages. When this limit is reached, the producer will block the messages for max.block.ms before raising an exception. If your batch size is more, allocate more memory to the producer buffer.

Additionally, to avoid keeping records queued indefinitely, you can set a timeout using request.timeout.ms. If this timeout expires before a message can be successfully sent, then it will be removed from the queue and an exception will be thrown.

  • acks: This configuration helps in configuring when producer will receive acknowledgment from the leader before considering that the message is committed successfully:
    • acks=0: Producer will not wait for any acknowledgment from the server. Producer will not know if the message is lost at any point in time and is not committed by the leader broker. Note that no retry will happen in this case and the message will be completely lost. This can be used when you want to achieve very high throughput and when you don't care about potential message loss.
    • acks=1: Producer will receive an acknowledgment as soon as the leader has written the message to its local log. If the leader fails to write the message to its log, producer will retry sending the data according to the retry policy set and avoid potential loss of messages. However, we can still have message loss in a scenario where the leader acknowledges to producer but does not replicate the message to the other broker before it goes down.
    • acks=all: Producer will only receive acknowledgment when the leader has received acknowledgment for all the replicas successfully. This is a safe setting where we cannot lose data if the replica number is sufficient to avoid such failures. Remember, throughput will be lesser then the first two settings.
  • batch.size: This setting allows the producer to batch the messages based on the partition up to the configured amount of size. When the batch reaches the limit, all messages in the batch will be sent. However, it's not necessary that producer wait for the batch to be full. It sends the batch after a specific time interval without worrying about the number of messages in the batch.
  • linger.ms: This represents an amount of time that a producer should wait for additional messages before sending a current batch to the broker. Kafka producer waits for the batch to be full or the configured linger.ms time; if any condition is met, it will send the batch to brokers. Producer will wait till the configured amount of time in milliseconds for any additional messages to get added to the current batch.
  • compression.type: By default, producer sends uncompressed messages to brokers. When sending a single message, it will not make that much sense, but when we use batches, it's good to use compression to avoid network overhead and increase throughput. The available compressions are GZIP, Snappy, or LZ4. Remember that more batching would lead to better compression.
  • retires: If message sending fails, this represents the number of times producer will retry sending messages before it throws an exception. It is irrespective of reseeding a message after receiving an exception.
  • max.in.flight.requests.per.connection: This is the number of messages producer can send to brokers without waiting for a response. If you do not care about the order of the messages, then setting its value to more than 1 will increase throughput. However, ordering may change if you set it to more than 1 with retry enabled.
  • partitioner.class: If you want to use a custom partitioner for your producer, then this configuration allows you to set the partitioner class, which implements the partitioner interface.
  • timeout.ms: This is the amount of time a leader will wait for its followers to acknowledge the message before sending an error to producer. This setting will only help when acks is set to all.
主站蜘蛛池模板: 马关县| 昭苏县| 岱山县| 商都县| 娱乐| 龙岩市| 湖南省| 阳泉市| 广丰县| 汉中市| 肇庆市| 集安市| 黔江区| 腾冲县| 五原县| 阳山县| 葫芦岛市| 塔城市| 铜山县| 宣汉县| 临漳县| 平阴县| 垦利县| 阜新市| 永胜县| 隆尧县| 汉沽区| 泗水县| 永仁县| 彩票| 望奎县| 远安县| 通海县| 万荣县| 同德县| 镇江市| 西华县| 牡丹江市| 永定县| 清水河县| 临夏市|