I'm a +1 (non-binding) — This looks like it would have saved us a lot of pain in an issue we had to debug recently. I can't go into details, but figuring out how to achieve this effect gave me quite a headache. :)
On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xiongq...@gmail.com> wrote: > Hi all, > > Can I have one more vote on this KIP? > Any comment is appreciated. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag > > > Xiongqi (Wesley) Wu > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xiongq...@gmail.com> wrote: > > > Thanks Dong. > > I have updated the KIP. > > > > Xiongqi (Wesley) Wu > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <lindon...@gmail.com> wrote: > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding) > >> > >> One minor comment: it may be a bit better to clarify in the public > >> interface section that the value of the newly added metric is determined > >> based by applying that formula across all compactable segments. For > >> example: > >> > >> The maximum value of Math.max(now - > >> earliest_timestamp_in_ms_of_uncompacted_segment - max.compaction.lag.ms > , > >> 0)/1000 across all compactable partitions, where the > >> max.compaction.lag.ms > >> can be overridden on per-topic basis. > >> > >> > >> > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xiongq...@gmail.com> wrote: > >> > >> > Thanks Joel. > >> > Tracking the delay at second granularity makes sense > >> > I have updated KIP. > >> > > >> > Xiongqi (Wesley) Wu > >> > > >> > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jjkosh...@gmail.com> > wrote: > >> > > >> > > +1 with one suggestion on the proposed metric. You should probably > >> > include > >> > > the unit. So for e.g., max-compaction-delay-secs. > >> > > > >> > > Joel > >> > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xiongq...@gmail.com> > >> wrote: > >> > > > >> > > > bump > >> > > > Xiongqi (Wesley) Wu > >> > > > > >> > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xiongq...@gmail.com> > >> > wrote: > >> > > > > >> > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for > >> feedback. > >> > > > > Can I have more feedback or VOTE on this KIP? > >> > > > > > >> > > > > > >> > > > > Xiongqi (Wesley) Wu > >> > > > > > >> > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu < > xiongq...@gmail.com> > >> > > wrote: > >> > > > > > >> > > > >> Any other votes or comments? > >> > > > >> > >> > > > >> Xiongqi (Wesley) Wu > >> > > > >> > >> > > > >> > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu < > xiongq...@gmail.com > >> > > >> > > > wrote: > >> > > > >> > >> > > > >>> Yes, more votes and code review. > >> > > > >>> > >> > > > >>> Xiongqi (Wesley) Wu > >> > > > >>> > >> > > > >>> > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann > >> > > <br...@zendesk.com.invalid > >> > > > > > >> > > > >>> wrote: > >> > > > >>> > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP. > >> > > > >>>> > >> > > > >>>> Where do we go from here? More votes? > >> > > > >>>> > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe < > >> cmcc...@apache.org> > >> > > > >>>> wrote: > >> > > > >>>> > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote: > >> > > > >>>> > > Thank you for comments. I will use '0' for now. > >> > > > >>>> > > > >> > > > >>>> > > If we create topics through admin client in the future, > we > >> > might > >> > > > >>>> perform > >> > > > >>>> > > some useful checks. (but the assumption is all brokers in > >> the > >> > > same > >> > > > >>>> > cluster > >> > > > >>>> > > have the same default configurations value, otherwise,it > >> might > >> > > > >>>> still be > >> > > > >>>> > > tricky to do such cross validation check.) > >> > > > >>>> > > >> > > > >>>> > This isn't something that we might do in the future-- this > is > >> > > > >>>> something we > >> > > > >>>> > are doing now. We already have Create Topic policies which > >> are > >> > > > >>>> enforced by > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is > >> one > >> > of > >> > > > the > >> > > > >>>> > motivations for getting rid of direct ZK access-- making > sure > >> > that > >> > > > >>>> these > >> > > > >>>> > policies are applied. > >> > > > >>>> > > >> > > > >>>> > I agree that having different configurations on different > >> > brokers > >> > > > can > >> > > > >>>> be > >> > > > >>>> > confusing and frustrating . That's why more configurations > >> are > >> > > being > >> > > > >>>> made > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored > >> > centrally > >> > > > in > >> > > > >>>> ZK, > >> > > > >>>> > so they are the same on all brokers (modulo propagation > >> delays). > >> > > In > >> > > > >>>> any > >> > > > >>>> > case, this is a general issue, not specific to "create > >> topics". > >> > > > >>>> > > >> > > > >>>> > cheers, > >> > > > >>>> > Colin > >> > > > >>>> > > >> > > > >>>> > > >> > > > >>>> > > > >> > > > >>>> > > > >> > > > >>>> > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > >> > > > >>>> > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe < > >> > > cmcc...@apache.org > >> > > > > > >> > > > >>>> > wrote: > >> > > > >>>> > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we should > >> > probably > >> > > be > >> > > > >>>> > > > consistent with how segment.ms works, and just use 0. > >> > > > >>>> > > > > >> > > > >>>> > > > best, > >> > > > >>>> > > > Colin > >> > > > >>>> > > > > >> > > > >>>> > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote: > >> > > > >>>> > > > > OK thanks for that clarification. I see why you're > >> > > > uncomfortable > >> > > > >>>> > with 0 > >> > > > >>>> > > > now. > >> > > > >>>> > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in > >> > > > >>>> configuration > >> > > > >>>> > > > options. > >> > > > >>>> > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1 similarly > in > >> > that > >> > > > >>>> > scenario, > >> > > > >>>> > > > > because it favours the person thinking about setting > >> the > >> > > > >>>> > configurations, > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case > >> especially > >> > > > when > >> > > > >>>> the > >> > > > >>>> > > > context > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner > cadence. > >> > > > >>>> > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is > some > >> > > > >>>> uniquness in > >> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces > >> > > > configuration > >> > > > >>>> > > > consistency > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case that > >> > nobody > >> > > > >>>> would > >> > > > >>>> > ever > >> > > > >>>> > > > > notice. For someone to understand why everywhere else > >> -1 > >> > is > >> > > > >>>> used to > >> > > > >>>> > > > > disable, but here 0 is used, they would need to learn > >> > about > >> > > > >>>> > segment.ms > >> > > > >>>> > > > > having a 1ms minimum and then after learning would > >> think > >> > > "who > >> > > > >>>> cares > >> > > > >>>> > about > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :) > >> > > > >>>> > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious > >> which > >> > > way > >> > > > >>>> Colin > >> > > > >>>> > falls > >> > > > >>>> > > > > now. > >> > > > >>>> > > > > > >> > > > >>>> > > > > Don't want to spend more time on this though, It's > well > >> > into > >> > > > >>>> > > > bikeshedding > >> > > > >>>> > > > > territory now. :) > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu < > >> > > > xiongq...@gmail.com> > >> > > > >>>> > wrote: > >> > > > >>>> > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of segment.ms > >> (which > >> > is > >> > > > >>>> 1ms) to > >> > > > >>>> > > > force > >> > > > >>>> > > > > > roll an active segment. > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > > 0, > >> the > >> > > > >>>> minimum of > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used > to > >> > seal > >> > > > an > >> > > > >>>> > active > >> > > > >>>> > > > > > segment. > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current > >> > > > >>>> implementation > >> > > > >>>> > will > >> > > > >>>> > > > > > treat it as disabled. > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat > >> max.compaction.lag=0 > >> > the > >> > > > >>>> same as > >> > > > >>>> > > > > > max.compaction.lag=1. > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > There might be a reason why we set the minimum of > >> > > > segment.ms > >> > > > >>>> to 1, > >> > > > >>>> > > > and I > >> > > > >>>> > > > > > don't want to break this assumption. > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann > >> > > > >>>> > <br...@zendesk.com.invalid> > >> > > > >>>> > > > > > wrote: > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the condition is > >> met > >> > > > right? > >> > > > >>>> So > >> > > > >>>> > I'm > >> > > > >>>> > > > > > > struggling to understand the relevance of > >> segment.ms > >> > > > here. > >> > > > >>>> > Maybe an > >> > > > >>>> > > > > > > example > >> > > > >>>> > > > > > > would help my understanding: > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999 > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > >> > > > >>>> > > > > > > max.compaction.lag.ms=1 > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms the > >> topic > >> > > > >>>> should be > >> > > > >>>> > > > eligible > >> > > > >>>> > > > > > > for compaction when the log compaction thread > gets > >> > > around > >> > > > to > >> > > > >>>> > > > evaluating > >> > > > >>>> > > > > > the > >> > > > >>>> > > > > > > topic. > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > if we have > >> > > > >>>> > > > > > > segment.ms=9999999 > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > >> > > > >>>> > > > > > > max.compaction.lag.ms=0 > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms the > >> topic > >> > > > >>>> should be > >> > > > >>>> > > > eligible > >> > > > >>>> > > > > > > for compaction when the log compaction thread > gets > >> > > around > >> > > > to > >> > > > >>>> > > > evaluating > >> > > > >>>> > > > > > the > >> > > > >>>> > > > > > > topic. > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > In both of those cases the change would mean a > new > >> > > segment > >> > > > >>>> is > >> > > > >>>> > rolled > >> > > > >>>> > > > so > >> > > > >>>> > > > > > the > >> > > > >>>> > > > > > > new message would be part of the compaction > task. 0 > >> > and > >> > > 1 > >> > > > >>>> are > >> > > > >>>> > > > practically > >> > > > >>>> > > > > > > the same meaning since neither is providing an > >> actual > >> > > > >>>> guarantee > >> > > > >>>> > at > >> > > > >>>> > > > such > >> > > > >>>> > > > > > low > >> > > > >>>> > > > > > > MS settings, but effectively tying it to both the > >> > > > frequency > >> > > > >>>> of > >> > > > >>>> > the > >> > > > >>>> > > > log > >> > > > >>>> > > > > > > cleaner running and the priority of the given > topic > >> > > being > >> > > > >>>> the > >> > > > >>>> > highest > >> > > > >>>> > > > > > > priority of all topics that are evaluated for > >> cleaning > >> > > on > >> > > > >>>> the > >> > > > >>>> > next > >> > > > >>>> > > > cycle. > >> > > > >>>> > > > > > > You've captured that nuance with careful > "skipped" > >> > > wording > >> > > > >>>> in > >> > > > >>>> > the KIP > >> > > > >>>> > > > > > > here "controls > >> > > > >>>> > > > > > > the max time interval a message/segment can be > >> skipped > >> > > for > >> > > > >>>> log > >> > > > >>>> > > > > > compaction". > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And how is > it > >> > > > >>>> relating to > >> > > > >>>> > > > > > segment.ms > >> > > > >>>> > > > > > > ? > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use > >> > > > segment.ms > >> > > > >>>> > > > instead?" as > >> > > > >>>> > > > > > a > >> > > > >>>> > > > > > > kind of third option? > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu < > >> > > > >>>> xiongq...@gmail.com> > >> > > > >>>> > > > wrote: > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > > To make it clear, > >> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we > >> need to > >> > > > come > >> > > > >>>> up > >> > > > >>>> > with > >> > > > >>>> > > > the > >> > > > >>>> > > > > > > > meaning of "0". > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but the > actual > >> > > > >>>> compaction > >> > > > >>>> > lag > >> > > > >>>> > > > will > >> > > > >>>> > > > > > be > >> > > > >>>> > > > > > > > segment.ms. > >> > > > >>>> > > > > > > > It has longer lag than setting the value to be > >> half > >> > of > >> > > > >>>> > segment.ms. > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag. > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > Here are two options. > >> > > > >>>> > > > > > > > Option 1: > >> > > > >>>> > > > > > > > Keep 0 as disabled > >> > > > >>>> > > > > > > > Option 2: > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = > >> segment.ms), > >> > > and > >> > > > >>>> > others. > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann > >> > > > >>>> > > > <br...@zendesk.com.invalid> > >> > > > >>>> > > > > > > > wrote: > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with these > >> settings > >> > > for > >> > > > >>>> > example: > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes > >> > > > >>>> > > > > > > > > queued.max.request.bytes > >> > > > >>>> > > > > > > > > retention.bytes > >> > > > >>>> > > > > > > > > retention.ms > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > and acks. > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS defaults, > >> max > >> > > > (acks), > >> > > > >>>> > etc. I > >> > > > >>>> > > > don't > >> > > > >>>> > > > > > > see > >> > > > >>>> > > > > > > > > much convention of 0 meaning those things. > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion > >> there > >> > is > >> > > > >>>> NULL is > >> > > > >>>> > used > >> > > > >>>> > > > > > where > >> > > > >>>> > > > > > > > > there's another setting in the hierarchy that > >> > would > >> > > be > >> > > > >>>> used > >> > > > >>>> > > > instead. > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann < > >> > > > >>>> > br...@zendesk.com> > >> > > > >>>> > > > > > wrote: > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then > we're > >> not > >> > > > being > >> > > > >>>> > > > consistent > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw > out > >> -1 > >> > > for > >> > > > >>>> > > > consideration > >> > > > >>>> > > > > > > > > > again :) > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu > < > >> > > > >>>> > > > xiongq...@gmail.com> > >> > > > >>>> > > > > > > > wrote: > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is > merged. > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of " > >> segment.ms" > >> > > to > >> > > > >>>> 1, we > >> > > > >>>> > > > cannot > >> > > > >>>> > > > > > set > >> > > > >>>> > > > > > > " > >> > > > >>>> > > > > > > > > >> segment.ms" to 0. > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> I also updated the title of this KIP. > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann > >> > > > >>>> > > > > > <br...@zendesk.com.invalid > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm > in > >> the > >> > > > >>>> minority. > >> > > > >>>> > :) > >> > > > >>>> > > > Can > >> > > > >>>> > > > > > we > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning > >> > disabled > >> > > > >>>> here: > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > >> > > > >>>> > > > > > < > >> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>> > >> > > > >>>> > > > > > > > < > >> > > https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > >> > > > >>>> > > > > > < > >> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>> > >> > > > >>>> > > > > > > > > >> < > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > >> > > > >>>> > > > > > < > >> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>> > >> > > > >>>> > > > > > > > < > >> > > https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > >> > > > >>>> > > > > > < > >> https://kafka.apache.org/documentation/#brokerconfigs > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> > ? > >> > > > >>>> > > > > > > > > >> > And while there it would be good if > >> > segment.ms > >> > > > is > >> > > > >>>> > > > documented > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some > >> > hierarchy > >> > > > of > >> > > > >>>> > configs > >> > > > >>>> > > > for > >> > > > >>>> > > > > > > that > >> > > > >>>> > > > > > > > > too > >> > > > >>>> > > > > > > > > >> > if its not set and null for others means > >> > > > disabled!) > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi > wu > >> < > >> > > > >>>> > > > xiongq...@gmail.com> > >> > > > >>>> > > > > > > > > wrote: > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate > >> > compaction, > >> > > > the > >> > > > >>>> > > > compaction > >> > > > >>>> > > > > > lag > >> > > > >>>> > > > > > > > is > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst > >> case. If > >> > > > >>>> segment.ms > >> > > > >>>> > is > >> > > > >>>> > > > 24 > >> > > > >>>> > > > > > > > hours, > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker > >> > guarantee > >> > > > than > >> > > > >>>> > setting > >> > > > >>>> > > > any > >> > > > >>>> > > > > > > > value > >> > > > >>>> > > > > > > > > >> less > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of > "max > >> > > > >>>> compaction > >> > > > >>>> > lag", > >> > > > >>>> > > > we > >> > > > >>>> > > > > > > > cannot > >> > > > >>>> > > > > > > > > >> have > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate > >> "disable". > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin > >> > McCabe < > >> > > > >>>> > > > > > cmcc...@apache.org > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett > >> Rann > >> > > > >>>> wrote: > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should > >> make > >> > 0 = > >> > > > >>>> > disable, to > >> > > > >>>> > > > be > >> > > > >>>> > > > > > > > > >> consistent > >> > > > >>>> > > > > > > > > >> > > with > >> > > > >>>> > > > > > > > > >> > > > > the other settings. > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable > and > >> > when > >> > > > >>>> seeing > >> > > > >>>> > it > >> > > > >>>> > > > in a > >> > > > >>>> > > > > > > > config > >> > > > >>>> > > > > > > > > >> it's > >> > > > >>>> > > > > > > > > >> > > > clear > >> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0 > >> doesn't > >> > > have > >> > > > >>>> to > >> > > > >>>> > mean > >> > > > >>>> > > > > > instant, > >> > > > >>>> > > > > > > > it > >> > > > >>>> > > > > > > > > >> just > >> > > > >>>> > > > > > > > > >> > > > means > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't > >> think 0 > >> > > is > >> > > > >>>> > intuitive > >> > > > >>>> > > > for > >> > > > >>>> > > > > > > > > disabled > >> > > > >>>> > > > > > > > > >> and > >> > > > >>>> > > > > > > > > >> > > it > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware > >> > > > segment.ms=0 > >> > > > >>>> == > >> > > > >>>> > > > > > disabled, > >> > > > >>>> > > > > > > > > but I > >> > > > >>>> > > > > > > > > >> > > think > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive. > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument for > >> keeping > >> > > > these > >> > > > >>>> two > >> > > > >>>> > > > > > > > configurations > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so > >> similar. I > >> > > > agree > >> > > > >>>> that > >> > > > >>>> > 0 > >> > > > >>>> > > > was an > >> > > > >>>> > > > > > > > > >> > unfortunate > >> > > > >>>> > > > > > > > > >> > > > choice., > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > best, > >> > > > >>>> > > > > > > > > >> > > > Colin > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM > >> Colin > >> > > > McCabe > >> > > > >>>> < > >> > > > >>>> > > > > > > > > cmcc...@apache.org> > >> > > > >>>> > > > > > > > > >> > > wrote: > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47, > >> > xiongqi > >> > > wu > >> > > > >>>> wrote: > >> > > > >>>> > > > > > > > > >> > > > > > > Colin, > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments. > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below. > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM > >> > Colin > >> > > > >>>> McCabe < > >> > > > >>>> > > > > > > > > >> cmcc...@apache.org> > >> > > > >>>> > > > > > > > > >> > > > wrote: > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi, > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP. > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit > >> ambiguous. > >> > Our > >> > > > >>>> > compaction > >> > > > >>>> > > > > > > policies > >> > > > >>>> > > > > > > > > are > >> > > > >>>> > > > > > > > > >> > > already > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It > >> seems > >> > > like > >> > > > >>>> this > >> > > > >>>> > > > change is > >> > > > >>>> > > > > > > > > focused > >> > > > >>>> > > > > > > > > >> > > around > >> > > > >>>> > > > > > > > > >> > > > > > adding > >> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms." > >> > Perhaps > >> > > > >>>> the KIP > >> > > > >>>> > > > title > >> > > > >>>> > > > > > > should > >> > > > >>>> > > > > > > > > be > >> > > > >>>> > > > > > > > > >> > > > something > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction > >> lag > >> > > > time"? > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will > >> change > >> > the > >> > > > >>>> title. > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is forced > >> to > >> > > roll > >> > > > >>>> when > >> > > > >>>> > > > either " > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms" > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" ( > >> log.roll.ms > >> > and > >> > > > >>>> > > > log.roll.hours) > >> > > > >>>> > > > > > > has > >> > > > >>>> > > > > > > > > >> > reached. > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the > max.compaction.lag.ms > >> is > >> > > low, > >> > > > >>>> it > >> > > > >>>> > seems > >> > > > >>>> > > > like > >> > > > >>>> > > > > > > > > segments > >> > > > >>>> > > > > > > > > >> > will > >> > > > >>>> > > > > > > > > >> > > > be > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This > >> can > >> > > be a > >> > > > >>>> > source of > >> > > > >>>> > > > > > > problems > >> > > > >>>> > > > > > > > > in > >> > > > >>>> > > > > > > > > >> the > >> > > > >>>> > > > > > > > > >> > > > > > cluster, > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many > different > >> > small > >> > > > log > >> > > > >>>> > segments > >> > > > >>>> > > > > > > > consumes > >> > > > >>>> > > > > > > > > a > >> > > > >>>> > > > > > > > > >> > huge > >> > > > >>>> > > > > > > > > >> > > > > > amount of > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources. > Therefore, > >> I > >> > > would > >> > > > >>>> > suggest > >> > > > >>>> > > > > > adding a > >> > > > >>>> > > > > > > > > >> > > broker-level > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows > us > >> to > >> > > set > >> > > > a > >> > > > >>>> > minimum > >> > > > >>>> > > > value > >> > > > >>>> > > > > > > for > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If > we > >> let > >> > > > >>>> users set > >> > > > >>>> > it > >> > > > >>>> > > > on a > >> > > > >>>> > > > > > > > > >> per-topic > >> > > > >>>> > > > > > > > > >> > > > basis, > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value of > >> 1 ms > >> > > or > >> > > > >>>> > something, > >> > > > >>>> > > > and > >> > > > >>>> > > > > > > > cause > >> > > > >>>> > > > > > > > > >> > chaos. > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to > >> > > > segment.ms > >> > > > >>>> as > >> > > > >>>> > well. > >> > > > >>>> > > > > > Today > >> > > > >>>> > > > > > > > > users > >> > > > >>>> > > > > > > > > >> can > >> > > > >>>> > > > > > > > > >> > > > set " > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low > >> value, > >> > and > >> > > > >>>> cause a > >> > > > >>>> > > > frequent > >> > > > >>>> > > > > > > > > >> rolling of > >> > > > >>>> > > > > > > > > >> > > > active > >> > > > >>>> > > > > > > > > >> > > > > > > segments. > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi, > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an existing > >> > > problem > >> > > > >>>> with > >> > > > >>>> > > > > > segment.ms. > >> > > > >>>> > > > > > > > > >> However, > >> > > > >>>> > > > > > > > > >> > > > that > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't > fix > >> > it. > >> > > As > >> > > > >>>> you > >> > > > >>>> > noted, > >> > > > >>>> > > > > > there > >> > > > >>>> > > > > > > > > will > >> > > > >>>> > > > > > > > > >> be > >> > > > >>>> > > > > > > > > >> > > more > >> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level > >> > retention > >> > > > >>>> > settings as > >> > > > >>>> > > > a > >> > > > >>>> > > > > > > result > >> > > > >>>> > > > > > > > > of > >> > > > >>>> > > > > > > > > >> > GDPR. > >> > > > >>>> > > > > > > > > >> > > > It > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing > >> > problems > >> > > > >>>> will > >> > > > >>>> > cause > >> > > > >>>> > > > more > >> > > > >>>> > > > > > > > > trouble. > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively > >> > > straightforward > >> > > > >>>> here -- > >> > > > >>>> > > > add a > >> > > > >>>> > > > > > > > > >> broker-level > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that > overrides > >> > > > >>>> per-topic > >> > > > >>>> > > > minimums. > >> > > > >>>> > > > > > We > >> > > > >>>> > > > > > > > can > >> > > > >>>> > > > > > > > > >> also > >> > > > >>>> > > > > > > > > >> > > fail > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message > when > >> > > someone > >> > > > >>>> > attempts > >> > > > >>>> > > > to > >> > > > >>>> > > > > > set > >> > > > >>>> > > > > > > an > >> > > > >>>> > > > > > > > > >> > invalid > >> > > > >>>> > > > > > > > > >> > > > > > configuration. > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of " > >> > > > >>>> > > > max.compaction.lag.ms" > >> > > > >>>> > > > > > > > should > >> > > > >>>> > > > > > > > > >> be > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of " > >> > segment.ms > >> > > ". > >> > > > >>>> Since > >> > > > >>>> > > > today the > >> > > > >>>> > > > > > > > > >> minimum of > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms" > >> also > >> > > > >>>> starts > >> > > > >>>> > with > >> > > > >>>> > > > 1. "0" > >> > > > >>>> > > > > > > > means > >> > > > >>>> > > > > > > > > >> > > > disable. I > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it > is > >> > hard > >> > > > to > >> > > > >>>> > define > >> > > > >>>> > > > the > >> > > > >>>> > > > > > > > meaning > >> > > > >>>> > > > > > > > > >> of 0 > >> > > > >>>> > > > > > > > > >> > > > because > >> > > > >>>> > > > > > > > > >> > > > > > we > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active > >> segment > >> > > > >>>> > immediately. > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should > >> make > >> > 0 = > >> > > > >>>> > disable, to > >> > > > >>>> > > > be > >> > > > >>>> > > > > > > > > >> consistent > >> > > > >>>> > > > > > > > > >> > > with > >> > > > >>>> > > > > > > > > >> > > > the > >> > > > >>>> > > > > > > > > >> > > > > > other settings. > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best, > >> > > > >>>> > > > > > > > > >> > > > > > Colin > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative > >> > > > >>>> configuration > >> > > > >>>> > is to > >> > > > >>>> > > > use > >> > > > >>>> > > > > > -1 > >> > > > >>>> > > > > > > > as > >> > > > >>>> > > > > > > > > >> > > > "disabled" > >> > > > >>>> > > > > > > > > >> > > > > > and > >> > > > >>>> > > > > > > > > >> > > > > > > > 0 > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction". > >> > > Because > >> > > > >>>> > compaction > >> > > > >>>> > > > lag > >> > > > >>>> > > > > > is > >> > > > >>>> > > > > > > > > still > >> > > > >>>> > > > > > > > > >> > > > determined > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on > min.compaction.lag > >> and > >> > > how > >> > > > >>>> long > >> > > > >>>> > to > >> > > > >>>> > > > roll > >> > > > >>>> > > > > > an > >> > > > >>>> > > > > > > > > active > >> > > > >>>> > > > > > > > > >> > > > segment, > >> > > > >>>> > > > > > > > > >> > > > > > > > the > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction > is > >> > > > >>>> undetermined > >> > > > >>>> > if > >> > > > >>>> > > > we > >> > > > >>>> > > > > > use > >> > > > >>>> > > > > > > > "0". > >> > > > >>>> > > > > > > > > >> On > >> > > > >>>> > > > > > > > > >> > the > >> > > > >>>> > > > > > > > > >> > > > other > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio" > >> > > > >>>> > > > > > > > to > >> > > > >>>> > > > > > > > > >> > achieve > >> > > > >>>> > > > > > > > > >> > > > the > >> > > > >>>> > > > > > > > > >> > > > > > > > same > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose > "0" > >> as > >> > > > >>>> "disabled". > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the > >> > invalid > >> > > > >>>> setting. > >> > > > >>>> > > > > > Treating > >> > > > >>>> > > > > > > 0 > >> > > > >>>> > > > > > > > > >> > > differently > >> > > > >>>> > > > > > > > > >> > > > > > than > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me. > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous > >> comment, I > >> > > am > >> > > > >>>> not > >> > > > >>>> > > > strongly > >> > > > >>>> > > > > > > > against, > >> > > > >>>> > > > > > > > > >> but > >> > > > >>>> > > > > > > > > >> > 0 > >> > > > >>>> > > > > > > > > >> > > is > >> > > > >>>> > > > > > > > > >> > > > > > not a > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my > >> option. > >> > So > >> > > I > >> > > > >>>> use > >> > > > >>>> > "0" as > >> > > > >>>> > > > > > > disabled > >> > > > >>>> > > > > > > > > >> state. > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best, > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at > 15:04, > >> > > > xiongqi > >> > > > >>>> wu > >> > > > >>>> > wrote: > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP. > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP: > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>> > >> > > > >>>> > > > > > > > > >> < > >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>>> > >> > > > >>>> > > > > > > > > >> > > < > >> > > > >>>> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>> > >> > > > >>>> > > > > > > > > >> < > >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>>>> > >> > > > >>>> > > > > > > > > >> > > > > > < > >> > > > >>>> > > > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>> > >> > > > >>>> > > > > > > > > >> < > >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>>> > >> > > > >>>> > > > > > > > > >> > > < > >> > > > >>>> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> >>> > >> > > > >>>> > > > > > > > > >> < > >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >> > > > >>>> > > > > > > > < > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > > > > > < > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > >> > >>>>>> > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > %3A+Time-based+log+compaction+policy > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation: > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > >> > > > >>>> > > > > > > > > >> < > https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>> > >> > > > >>>> > > > > > > > > >> > > < > >> https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > >> > > > >>>> > > > > > > > > >> < > https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>> > >> > > > >>>> > > > > > > > > >> > > > > > < > >> > > > https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > >> > > > >>>> > > > > > > > > >> < > https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>> > >> > > > >>>> > > > > > > > > >> > > < > >> https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > >> > > > >>>> > > > > > > > > >> < > https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>> > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > -- > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > Brett Rann > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000 > >> > > Australia > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017 > >> > > > >>>> > > > > > > > > >> > > >> > > > >>>> > > > > > > > > >> > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > -- > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > Brett Rann > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 > >> Australia > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017 > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > -- > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > Brett Rann > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 > >> Australia > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017 > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > >> > > > >>>> > > >> > > > >>>> > >> > > > >>> > >> > > > > >> > > > >> > > >> > > >
