Thanks Dongjin. So you're saying that reducing the buffer size didn't reduce the compression rate for codecs like lz4? If so, that would suggest reducing the default value, but that seems odd.
Ismael On Sat, Jun 5, 2021, 9:25 AM Dongjin Lee <dong...@apache.org> wrote: > Hello Kafka dev, > > I hope to reboot the discussion of KIP-390: Support Compression Level > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level > >. > It proposes to add a new option, 'compression.level', that controls the > compression level. > > This KIP has been submitted more than one year ago, but had been neglected > for a long time. Recently I reworked it from scratch with the following > differences: > > 1. Tested how it works with a real-world dataset. As you can see in the > updated KIP, *this feature can improve the producer's message/second rate > by more than 50%*, such a significant enhancement. > 2. Dropped 'compression.buffer.size' option that was in the initial work. > With the repeated benchmarks, I could not find any evidence this option > results in meaningful differences. So I removed it. > > All feedback will be highly appreciated. > > Best, > Dongjin > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* >
