Hi all, I have some questions about how brokers batch records to send to consumers.
One of the configuration properties for a consumer is fetch.max.bytes. Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about fetch.max.bytes : "This property lets you specify the maximum bytes that Kafka will return whenever the consumer polls a broker (50 MB by default). [...] Note that records are sent to the client in batches, and if the first record-batch that the broker has to send exceeds this size, the batch will be sent and the limit will be ignored." My questions: 1. Why can't the broker remove messages from the first batch? So that fetch.max.bytes is not violated. 2. How does the broker choose which partitions (from those that the consumer is assigned to) contribute messages to a response batch? 3. Why does the broker send records in batches to the consumer?
