I have a question related to batch.size producer configuration.
What happens when batch.size has reached and the producer app thread sends more data?
Does the thread block till space becomes available in the buffer containing the batch?
Best Answer
batch.size measures batch size in total bytes instead of the number of messages. It controls how many bytes of data to collect before sending messages to the Kafka broker. Set this as high as possible, without exceeding available memory. The default value is 16384.
When you use Producer.send(), you fill up buffers on the producer. When a buffer is full, the producer sends the buffer to the Kafka broker and begins to refill the buffer.