Re: Message batch & compression doesn't work as expected

2019-12-16 Thread Frank Zhou
Hi, I think the root cause of this is transaction. The producer we are using has enabled transaction, and batching then is not working as expected. Once we turn it off, the situation is back to normal, I can see batching is performed, and everything looks fine. Is transaction not working well with

Re: Message batch & compression doesn't work as expected

2019-12-12 Thread Frank Zhou
Hi, Definitely will check this out, thanks. We just started tuning recently, and we are quite new to kafka world. The problem we are facing is with batch.size and linger.ms both set, the patch sent out doesn't seem to meet either one condition. We tried with more "reasonable" values as well, it ju

Re: Message batch & compression doesn't work as expected

2019-12-12 Thread M. Manna
Frank, On Thu, 12 Dec 2019 at 11:28, Frank Zhou wrote: > Hi, > > I am testing kafka client on message batch and compression. I have enabled > message batching along with compression, with batch.size set to 3M, > linger.ms set to 5000ms and compression.type set to gzip(Attached whole > producer c