Thank you for writing the KIP Assane. In general, exposing a "pluggable" interface is not a decision made lightly because it limits our ability to remove / change that interface in future. Any future changes to the interface will have to remain compatible with existing plugins which limits the flexibility of changes we can make inside Kafka. Hence, we need a strong motivation for adding a pluggable interface.
1\ May I ask the motivation for this KIP? Are the current compression codecs (zstd, gzip, lz4, snappy) not sufficient for your use case? Would proving fine grained compression options as proposed in https://issues.apache.org/jira/browse/KAFKA-7632 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level address your use case? 2\ "This option impacts the following processes" -> This should also include the decompression and compression that occurs during message version transformation, i.e. when client send message with V1 and broker expects in V2, we convert the message and recompress it. -- Divij Vaidya On Mon, Dec 18, 2023 at 7:22 PM Diop, Assane <assane.d...@intel.com> wrote: > I would like to bring some attention to this KIP. We have added an > interface to the compression code that allow anyone to build their own > compression plugin and integrate easily back to kafka. > > Assane > > -----Original Message----- > From: Diop, Assane <assane.d...@intel.com> > Sent: Monday, October 2, 2023 9:27 AM > To: dev@kafka.apache.org > Subject: DISCUSS KIP-984 Add pluggable compression interface to Kafka > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-984%3A+Add+pluggable+compression+interface+to+Kafka >
