Hi, Luis, Thanks for the reply. A few more comments below.
1. About the topic level configuration. It seems that it's useful for the new configs to be at the topic level. Currently, the following configs related to compaction are already at the topic level. min.cleanable.dirty.ratio min.compaction.lag.ms cleanup.policy 2. Have you documented the memory impact in the KIP? 3. Could you document how we deal with the last message in the log, which is potentially cleanable now? 4. Could you document how key deletion will be handled? 10. As for Jason's proposal on CompactionStrategy, it does make the feature more general. On the other hand, it will be useful not to require user-level code if the compaction value only comes from the header. 20. "If compaction.strategy.header is chosen and compaction.strategy.header is not set, the KIP falls back to offset." I am wondering if it's better to just fail the configuration in the case. Jun On Thu, Aug 16, 2018 at 1:33 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Regarding "broker-agnostic of headers": there are some KIPs from Streams to > use headers for internal purposes as well, e.g. KIP-258 and KIP-213 (I > admit there may be a conflict with user space, but practically I think it > is very rare). So I think we are very likely going to make Kafka internals > to be "headers-aware" anyways. > > Regarding the general API: I think it is a good idea in general, but it may > still have limits: note that right now our KIP enforce a header type to be > long, and we have a very careful discussion about the fall-back policy if > header does not have the specified key or if the value is not long-typed; > but if we enforce long type version in the interface, it would require > users trying to customizing their compaction logic (think: based on some > value payload field) to transform their fields to long as well. So I feel > we can still go with the current proposed approach, and only consider this > general API if we observe it does have a general usage requirement. By that > time we can still extend the config values of "log.cleaner.compaction. > strategy" to "offset, timestamp, header, myFuncName". > > @Bertus > > Thanks for your feedback, I believe the proposed config is indeed for both > global (for the whole broker) and per-topic, Luís can confirm if this is > the case, and update the wiki page to make it clear. > > > Guozhang > > > On Thu, Aug 16, 2018 at 9:09 AM, Bertus Greeff < > bgre...@microsoft.com.invalid> wrote: > > > I'm interested to know the status of this KIP. I see that the status is > > "Voting". How long does this normally take? > > > > We want to use Kafka and this KIP provides exactly the log compaction > > logic that we want for many of our projects. > > > > One piece of feedback that I have is that log.cleaner.compaction. > strategy > > and log.cleaner.compaction.strategy.header needs to be per topic. The > > text of the KIP makes it sound that the config is only available for all > > topics but this makes no sense. Different topics will need different > > strategies and/or headers. > > > > From the KIP: > > Provide the configuration for the individual topics > > None of the configurations for log compaction are available at topic > > level, so adding it there is not a part of this KIP > > > > > > > > On 2018/04/05 08:44:00, Luís Cabral <l...@yahoo.com.INVALID> wrote: > > > Hello all,> > > > Starting a discussion for this feature.> > > > KIP-280 : https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 280% > > 3A+Enhanced+log+compactionPull-4822 : https://github.com/apache/kafk > > a/pull/4822> > > > > > Kind Regards,Luís> > > > > > > -- > -- Guozhang >
