Oh, that makes a lot more sense! I assumed that the batch size was in terms of number of messages, not the number of bytes because it was so small. What would be a reasonable value to use? Will 1-2 MB be too large and bursty?
On Thu, Feb 26, 2015 at 10:07 AM, Harsha <ka...@harsha.io> wrote: > Akshat, > Produce.batch_size is in bytes and if your messages avg size is > 310 bytes and your current number of messages per batch is 46 you > are getting close to the max batch size 16384. Did you try > increasing the producer batch_size bytes? > -Harsha > > On Thu, Feb 26, 2015, at 09:49 AM, Akshat Aranya wrote: > > Hi, > > > > I am using the new Producer API in Kafka 0.8.2. I am writing messages to > > Kafka that are ~310 bytes long with the same partition key to one single > > . > > I'm mostly using the default Producer config, which sets the max batch > > size > > to 16,384. However, looking at the JMX stats on the broker side, I see > > that I'm only getting an average batch size of 46. I also tried > > increasing > > the linger.ms value to 100ms (default is 0), but that didn't help > either. > > Is there something else that I can tune that will improve write batching? > > > > Thanks, > > Akshat >
