Hey Brett, I'd suggest going ahead with a KIP if you already have some ideas about it. One of the complications you may run into is that the active segment is not cleaned at present.
I went ahead and gave you wiki permission. I assumed your username is brettrann, so let me know if it's different. Thanks, Jason On Wed, Aug 1, 2018 at 5:38 PM, Brett Rann <br...@zendesk.com.invalid> wrote: > Ping on this. We're prepared to do the work i'm just not sure of the right > way to start. If nobody wants to discuss it now should we just write up a > KIP? > > On Thu, Jul 19, 2018 at 10:57 AM Brett Rann <br...@zendesk.com> wrote: > > > re: https://issues.apache.org/jira/browse/KAFKA-7137 > > > > My team is investigating what would be involved in code changes to give > > some finer control over when compaction runs. Detail in the ticket but > > essentially the current way is that dirty.ratio is hijacked to be set to > 0 > > to give guarantees that a tombstone record will result in previous record > > being deleted within a set time. > > > > This results in a lot of unnecessary compaction, and we would like to add > > the ability to provide a max delay for compaction to be used in > combination > > with a "healthy" dirty.ratio setting to meet this requirement instead. > > > > And to consider an API to trigger compaction (but that is looking > > difficult because compaction isn't really triggered, it's evaluated by > the > > log cleaner thread). > > > > We'd like to get some discussion going around this. Should we draft up a > > KIP first then kick off a discussion? (if yes, can I get KIP access for: > > brettr...@gmail.com ) > > > > Thanks! > > > > -- > > > > Brett Rann > > > > Senior DevOps Engineer > > > > > > Zendesk International Ltd > > > > 395 Collins Street, Melbourne VIC 3000 Australia > > > > Mobile: +61 (0) 418 826 017 > > > > > -- > > Brett Rann > > Senior DevOps Engineer > > > Zendesk International Ltd > > 395 Collins Street, Melbourne VIC 3000 Australia > > Mobile: +61 (0) 418 826 017 >
