Thanks Jason. >From what we've seen so far adding a max time should be the easy part because there's already a block of checks for if compaction should be run. The active segment issue can be dealt with by specifying a time based segment.
Triggering compaction is the harder part given it's not really triggered but evaluated by the log cleaner. We might just go with the config setting but will look further into triggering. On Thu, Aug 2, 2018 at 10:56 AM Jason Gustafson <ja...@confluent.io> wrote: > Hey Brett, > > I'd suggest going ahead with a KIP if you already have some ideas about it. > One of the complications you may run into is that the active segment is not > cleaned at present. > > I went ahead and gave you wiki permission. I assumed your username is > brettrann, so let me know if it's different. > > Thanks, > Jason > > On Wed, Aug 1, 2018 at 5:38 PM, Brett Rann <br...@zendesk.com.invalid> > wrote: > > > Ping on this. We're prepared to do the work i'm just not sure of the > right > > way to start. If nobody wants to discuss it now should we just write up a > > KIP? > > > > On Thu, Jul 19, 2018 at 10:57 AM Brett Rann <br...@zendesk.com> wrote: > > > > > re: https://issues.apache.org/jira/browse/KAFKA-7137 > <https://issues.apache.org/jira/browse/KAFKA-7137> > > > > > > My team is investigating what would be involved in code changes to give > > > some finer control over when compaction runs. Detail in the ticket but > > > essentially the current way is that dirty.ratio is hijacked to be set > to > > 0 > > > to give guarantees that a tombstone record will result in previous > record > > > being deleted within a set time. > > > > > > This results in a lot of unnecessary compaction, and we would like to > add > > > the ability to provide a max delay for compaction to be used in > > combination > > > with a "healthy" dirty.ratio setting to meet this requirement instead. > > > > > > And to consider an API to trigger compaction (but that is looking > > > difficult because compaction isn't really triggered, it's evaluated by > > the > > > log cleaner thread). > > > > > > We'd like to get some discussion going around this. Should we draft up > a > > > KIP first then kick off a discussion? (if yes, can I get KIP access > for: > > > brettr...@gmail.com ) > > > > > > Thanks! > > > > > > -- > > > > > > Brett Rann > > > > > > Senior DevOps Engineer > > > > > > > > > Zendesk International Ltd > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia > > > > > > Mobile: +61 (0) 418 826 017 > > > > > > > > > -- > > > > Brett Rann > > > > Senior DevOps Engineer > > > > > > Zendesk International Ltd > > > > 395 Collins Street, Melbourne VIC 3000 Australia > > > > Mobile: +61 (0) 418 826 017 > > > -- Brett Rann Senior DevOps Engineer Zendesk International Ltd 395 Collins Street, Melbourne VIC 3000 Australia Mobile: +61 (0) 418 826 017
