Hi Mickael, +1 to getting KIP-390 merged with the per-codec level config.
Thanks, Tom On Fri, 9 Feb 2024 at 07:27, Diop, Assane <assane.d...@intel.com> wrote: > > Hi Mickael > I will take some time today to add some documentation to KIP-984. I actually > have a working implementation of the plugin and I have been running test. I > would be ready to do a PR anytime. > As far as motivation, Divij did asked about that and I responded in an email > to the mailing list but I will update the KIP -984 accordingly as well. > > Assane > > -----Original Message----- > From: Mickael Maison <mickael.mai...@gmail.com> > Sent: Wednesday, February 7, 2024 6:40 AM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-390: Support Compression Level (rebooted) > > Hi Divij, > > Thanks for bringing that point. After reading KIP-984, I don't think it > supersedes KIP-390/KIP-780. Being able to tune the built-in codecs would > directly benefit many users. It may also cover some scenarios that motivated > KIP-984 without requiring users to write a custom codec. > I've not commented in the KIP-984 thread yet but at the moment it seems very > light on details (no proposed API for codecs, no explanations of error > scenarios if clients or brokers don't have compatible codecs), including the > motivation which is important when exposing new APIs. On the other hand, > KIP-390/KIP-780 have much more details with benchmarks to support the > motivation. > > In my opinion starting with the compression level (KIP-390) is a good first > step and I think we should focus on that and deliver it. I believe one of the > reasons KIP-780 wasn't voted is because we never delivered KIP-390 and nobody > was keen on building a KIP on top of another undelivered KIP. > > Thanks, > Mickael > > > On Wed, Feb 7, 2024 at 12:27 PM Divij Vaidya <divijvaidy...@gmail.com> wrote: > > > > Hey Mickael > > > > Since this KIP was written, we have a new proposal to make the > > compression completely pluggable > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-984%3A+Add+pluggable+compression+interface+to+Kafka. > > If we implement that KIP, would it supersede the need for adding fine > > grain compression controls in Kafka? > > > > It might be beneficial to have a joint proposal of these two KIPs > > which may satisfy both use cases. > > > > -- > > Divij Vaidya > > > > > > > > On Wed, Feb 7, 2024 at 11:14 AM Mickael Maison > > <mickael.mai...@gmail.com> > > wrote: > > > > > Hi, > > > > > > I'm resurrecting this old thread as this KIP would be a nice > > > improvement and almost 3 years later the PR for this KIP has still > > > not been merged! > > > > > > The reason is that during reviews we noticed the proposed > > > configuration, compression.level, was not easy to use as each codec > > > has its own valid range of levels [0]. > > > > > > As proposed by Jun in the PR [1], I updated the KIP to use > > > compression.<codec>.level configurations instead of a single > > > compression.level setting. This syntax would also line up with the > > > proposal to add per-codec configuration options from KIP-780 [2] > > > (still to be voted). I moved the original proposal to the rejected > > > section. > > > > > > I've put the original voters and KIP author on CC. Let me know if > > > you have any feedback. > > > > > > 0: https://github.com/apache/kafka/pull/10826 > > > 1: > > > https://github.com/apache/kafka/pull/10826#issuecomment-1795952612 > > > 2: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support > > > +fine-grained+compression+options > > > > > > Thanks, > > > Mickael > > > > > > > > > On Fri, Jun 11, 2021 at 10:00 AM Dongjin Lee <dong...@apache.org> wrote: > > > > > > > > This KIP is now passed with: > > > > > > > > - binding: +3 (Ismael, Tom, Konstantine) > > > > - non-binding: +1 (Ryanne) > > > > > > > > Thanks again to all the supporters. I also updated the KIP by > > > > moving the compression buffer option into the 'Future Works' > > > > section, as Ismael proposed. > > > > > > > > Best, > > > > Dongjin > > > > > > > > > > > > > > > > On Fri, Jun 11, 2021 at 3:03 AM Konstantine Karantasis > > > > <konstant...@confluent.io.invalid> wrote: > > > > > > > > > Makes sense. Looks like a good improvement. Thanks for including > > > > > the evaluation in the proposal Dongjin. > > > > > > > > > > +1 (binding) > > > > > > > > > > Konstantine > > > > > > > > > > On Wed, Jun 9, 2021 at 6:59 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > > > > > > > Thanks Ismel, Tom and Ryanne, > > > > > > > > > > > > I am now updating the KIP about the further works. Sure, You > > > > > > won't be disappointed. > > > > > > > > > > > > As of Present: > > > > > > > > > > > > - binding: +2 (Ismael, Tom) > > > > > > - non-binding: +1 (Ryanne) > > > > > > > > > > > > Anyone else? > > > > > > > > > > > > Best, > > > > > > Dongjin > > > > > > > > > > > > On Thu, Jun 10, 2021 at 2:03 AM Tom Bentley > > > > > > <tbent...@redhat.com> > > > wrote: > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > Thanks for the KIP, +1 (binding). > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > On Wed, Jun 9, 2021 at 5:16 PM Ismael Juma > > > > > > > <ism...@juma.me.uk> > > > wrote: > > > > > > > > > > > > > > > I'm +1 on the proposed change. As I stated in the discuss > > > thread, I > > > > > > don't > > > > > > > > think we should rule out the buffer size config, but we > > > > > > > > could > > > list > > > > > that > > > > > > > as > > > > > > > > future work vs rejected alternatives. > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > On Sat, Jun 5, 2021 at 2:37 PM Dongjin Lee > > > > > > > > <dong...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I'd like to open a voting thread for KIP-390: Support > > > Compression > > > > > > Level > > > > > > > > > (rebooted): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support > > > +Compression+Level > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: > > > > > > > > > <http://goog_969573159/>github.com/dongjinleekr > > > > > > > > > <https://github.com/dongjinleekr>keybase: > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Dongjin Lee* > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > <https://github.com/dongjinleekr>keybase: > > > > > https://keybase.io/dongjinleekr > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > > > speakerdeck.com/dongjin > > > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <https://github.com/dongjinleekr>keybase: > > > https://keybase.io/dongjinleekr > > > > <https://keybase.io/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > speakerdeck.com/dongjin > > > > <https://speakerdeck.com/dongjin>* > > >
