I would like to follow up with this more concrete question: For the purpose of achieving decent compression ratios, I am having difficulty finding information on whether the broker can perform additional record batching beyond the batching received from a producer. It seems like in a low-latency application where the producer recorder buffer and linger are small, it is difficult or impossible to achieve effective compression on the broker side, because the producer sending very small batches.
So, am I correct in my understand that the broker appends record batches exactly as they are received from the producer? Or can it do its own batching to improve compression? Thank you, Dmitry On Sun, Dec 10, 2017 at 7:44 PM, Dmitry Minkovsky <dminkov...@gmail.com> wrote: > This is hopefully my final question for a while. > > I noticed that compression is disabled by default. Why is this? My best > guess is that compression doesn't work well for short messages > <https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch>, > which was maybe identified as the majority use-case for Kafka. But, > producers batch records based on buffer/linger, and in my understanding the > whole record batch is compressed together. > > So, what's the deal? Should I turn on compression in production? Does it > depends on my anticipated batch size? I am using Kafka Streams with a very > low linger, so most of my batches will likely be very small. > > Thank you! > > >
