Yes. we want to enforce a max time interval from a message arrival time to the time the corresponding log segment needs to be compacted.
Today, if the message arriving rate is low for a log compacted topic, the dirty ratio increases very slowly. As a result, a log segment might be un-compacted for a long time. Xiongqi (Welsey) Wu On Mon, Aug 13, 2018 at 2:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Guess I need to carefully read the wiki page before asking :) Thanks! > > Another qq after reading the proposal: is it a complimentary to KIP-58 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 58+-+Make+Log+Compaction+Point+Configurable), > just that KIP-58 is a "upper-bound" on what messages can be compacted, and > this is for a "lower-bound" on what messages NEED to be compacted? > > > Guozhang > > On Mon, Aug 13, 2018 at 2:31 PM, xiongqi wu <xiongq...@gmail.com> wrote: > > > HI Guozhang, > > > > As I mentioned in the motivation section, KIP-280 focuses on how to > compact > > the log segment to resolve the out of order messages compaction issue. > > The issue we try to address in this KIP is different: we want to > introduce > > a compaction policy so that a log segment can be pickup for compaction > > after a specified time interval. One use case is for GDPR to ensure > timely > > deletion of user record. > > > > There is no conflict and overlapping between this KIP and KIP-280. > > > > Thank you! > > > > > > On Mon, Aug 13, 2018 at 1:33 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hello Xiongqi, > > > > > > I think this KIP is already been covered in KIP-280? Could you check > out > > > that one and see if it is the case. > > > > > > > > > Guozhang > > > > > > > > > On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu <xiongq...@gmail.com> > wrote: > > > > > > > Hi Kafka, > > > > > > > > Just updated the confluence page to include the link to this KIP. > > > > > > > > Any comment will be appreciated: > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A > > > > +Time-based+log+compaction+policy > > > > > > > > Thank you. > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu <xiongq...@gmail.com> > > wrote: > > > > > > > > > Hi Kafka, > > > > > > > > > > This KIP tries to address GDPR concern to fulfill deletion request > on > > > > time > > > > > through time-based log compaction on a compaction enabled topic: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 354%3A+Time-based+log+compaction+policy > > > > > > > > > > Any feedback will be appreciated. > > > > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
