Hi Xiongqi, Thinking about this a little bit more, it seems like we don't have any guarantees just by looking at the timestamp of the first message in a log segment. Similarly, we don't have any guarantees just by looking at the maxTimestamp of the previous log segment. Old data could appear anywhere-- you could put data that was years old in the middle of a segment from 2018.
However, if log.message.timestamp.difference.max.ms is set, then we can make some actual guarantees that old data gets purged-- which is what the GDPR requires, of course. Overall, maybe we can make KIP-354 simpler by just always looking at the timestamp of the first log message. I don't think looking at the maxTimestamp of the previous segment is any more accurate. Aside from that, looks good, since we can get what we need with the combination of this and log.message.timestamp.difference.max.ms. best, Colin On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote: > Thanks for binding and non-binding votes. > Can I get one more binding vote? > > Thanks in advance! > > Xiongqi (Wesley) Wu > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <m...@frmr.me> wrote: > > > I'm a +1 (non-binding) — This looks like it would have saved us a lot of > > pain in an issue we had to debug recently. I can't go into details, but > > figuring out how to achieve this effect gave me quite a headache. :) > > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xiongq...@gmail.com> wrote: > > > > > Hi all, > > > > > > Can I have one more vote on this KIP? > > > Any comment is appreciated. > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xiongq...@gmail.com> wrote: > > > > > > > Thanks Dong. > > > > I have updated the KIP. > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <lindon...@gmail.com> wrote: > > > > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding) > > > >> > > > >> One minor comment: it may be a bit better to clarify in the public > > > >> interface section that the value of the newly added metric is > > determined > > > >> based by applying that formula across all compactable segments. For > > > >> example: > > > >> > > > >> The maximum value of Math.max(now - > > > >> earliest_timestamp_in_ms_of_uncompacted_segment - > > max.compaction.lag.ms > > > , > > > >> 0)/1000 across all compactable partitions, where the > > > >> max.compaction.lag.ms > > > >> can be overridden on per-topic basis. > > > >> > > > >> > > > >> > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xiongq...@gmail.com> > > wrote: > > > >> > > > >> > Thanks Joel. > > > >> > Tracking the delay at second granularity makes sense > > > >> > I have updated KIP. > > > >> > > > > >> > Xiongqi (Wesley) Wu > > > >> > > > > >> > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jjkosh...@gmail.com> > > > wrote: > > > >> > > > > >> > > +1 with one suggestion on the proposed metric. You should probably > > > >> > include > > > >> > > the unit. So for e.g., max-compaction-delay-secs. > > > >> > > > > > >> > > Joel > > > >> > > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xiongq...@gmail.com> > > > >> wrote: > > > >> > > > > > >> > > > bump > > > >> > > > Xiongqi (Wesley) Wu > > > >> > > > > > > >> > > > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xiongq...@gmail.com > > > > > > >> > wrote: > > > >> > > > > > > >> > > > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for > > > >> feedback. > > > >> > > > > Can I have more feedback or VOTE on this KIP? > > > >> > > > > > > > >> > > > > > > > >> > > > > Xiongqi (Wesley) Wu > > > >> > > > > > > > >> > > > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu < > > > xiongq...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > > >> > > > >> Any other votes or comments? > > > >> > > > >> > > > >> > > > >> Xiongqi (Wesley) Wu > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu < > > > xiongq...@gmail.com > > > >> > > > > >> > > > wrote: > > > >> > > > >> > > > >> > > > >>> Yes, more votes and code review. > > > >> > > > >>> > > > >> > > > >>> Xiongqi (Wesley) Wu > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann > > > >> > > <br...@zendesk.com.invalid > > > >> > > > > > > > >> > > > >>> wrote: > > > >> > > > >>> > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP. > > > >> > > > >>>> > > > >> > > > >>>> Where do we go from here? More votes? > > > >> > > > >>>> > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe < > > > >> cmcc...@apache.org> > > > >> > > > >>>> wrote: > > > >> > > > >>>> > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote: > > > >> > > > >>>> > > Thank you for comments. I will use '0' for now. > > > >> > > > >>>> > > > > > >> > > > >>>> > > If we create topics through admin client in the future, > > > we > > > >> > might > > > >> > > > >>>> perform > > > >> > > > >>>> > > some useful checks. (but the assumption is all brokers > > in > > > >> the > > > >> > > same > > > >> > > > >>>> > cluster > > > >> > > > >>>> > > have the same default configurations value, > > otherwise,it > > > >> might > > > >> > > > >>>> still be > > > >> > > > >>>> > > tricky to do such cross validation check.) > > > >> > > > >>>> > > > > >> > > > >>>> > This isn't something that we might do in the future-- > > this > > > is > > > >> > > > >>>> something we > > > >> > > > >>>> > are doing now. We already have Create Topic policies > > which > > > >> are > > > >> > > > >>>> enforced by > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This > > is > > > >> one > > > >> > of > > > >> > > > the > > > >> > > > >>>> > motivations for getting rid of direct ZK access-- making > > > sure > > > >> > that > > > >> > > > >>>> these > > > >> > > > >>>> > policies are applied. > > > >> > > > >>>> > > > > >> > > > >>>> > I agree that having different configurations on different > > > >> > brokers > > > >> > > > can > > > >> > > > >>>> be > > > >> > > > >>>> > confusing and frustrating . That's why more > > configurations > > > >> are > > > >> > > being > > > >> > > > >>>> made > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored > > > >> > centrally > > > >> > > > in > > > >> > > > >>>> ZK, > > > >> > > > >>>> > so they are the same on all brokers (modulo propagation > > > >> delays). > > > >> > > In > > > >> > > > >>>> any > > > >> > > > >>>> > case, this is a general issue, not specific to "create > > > >> topics". > > > >> > > > >>>> > > > > >> > > > >>>> > cheers, > > > >> > > > >>>> > Colin > > > >> > > > >>>> > > > > >> > > > >>>> > > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > >> > > > >>>> > > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe < > > > >> > > cmcc...@apache.org > > > >> > > > > > > > >> > > > >>>> > wrote: > > > >> > > > >>>> > > > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we should > > > >> > probably > > > >> > > be > > > >> > > > >>>> > > > consistent with how segment.ms works, and just use > > 0. > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > best, > > > >> > > > >>>> > > > Colin > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote: > > > >> > > > >>>> > > > > OK thanks for that clarification. I see why you're > > > >> > > > uncomfortable > > > >> > > > >>>> > with 0 > > > >> > > > >>>> > > > now. > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in > > > >> > > > >>>> configuration > > > >> > > > >>>> > > > options. > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1 > > similarly > > > in > > > >> > that > > > >> > > > >>>> > scenario, > > > >> > > > >>>> > > > > because it favours the person thinking about > > setting > > > >> the > > > >> > > > >>>> > configurations, > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case > > > >> especially > > > >> > > > when > > > >> > > > >>>> the > > > >> > > > >>>> > > > context > > > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner > > > cadence. > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is > > > some > > > >> > > > >>>> uniquness in > > > >> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces > > > >> > > > configuration > > > >> > > > >>>> > > > consistency > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case > > that > > > >> > nobody > > > >> > > > >>>> would > > > >> > > > >>>> > ever > > > >> > > > >>>> > > > > notice. For someone to understand why everywhere > > else > > > >> -1 > > > >> > is > > > >> > > > >>>> used to > > > >> > > > >>>> > > > > disable, but here 0 is used, they would need to > > learn > > > >> > about > > > >> > > > >>>> > segment.ms > > > >> > > > >>>> > > > > having a 1ms minimum and then after learning would > > > >> think > > > >> > > "who > > > >> > > > >>>> cares > > > >> > > > >>>> > about > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :) > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious > > > >> which > > > >> > > way > > > >> > > > >>>> Colin > > > >> > > > >>>> > falls > > > >> > > > >>>> > > > > now. > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > Don't want to spend more time on this though, It's > > > well > > > >> > into > > > >> > > > >>>> > > > bikeshedding > > > >> > > > >>>> > > > > territory now. :) > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu < > > > >> > > > xiongq...@gmail.com> > > > >> > > > >>>> > wrote: > > > >> > > > >>>> > > > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of segment.ms > > > >> (which > > > >> > is > > > >> > > > >>>> 1ms) to > > > >> > > > >>>> > > > force > > > >> > > > >>>> > > > > > roll an active segment. > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > > > > 0, > > > >> the > > > >> > > > >>>> minimum of > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be > > used > > > to > > > >> > seal > > > >> > > > an > > > >> > > > >>>> > active > > > >> > > > >>>> > > > > > segment. > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the > > current > > > >> > > > >>>> implementation > > > >> > > > >>>> > will > > > >> > > > >>>> > > > > > treat it as disabled. > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat > > > >> max.compaction.lag=0 > > > >> > the > > > >> > > > >>>> same as > > > >> > > > >>>> > > > > > max.compaction.lag=1. > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > There might be a reason why we set the minimum of > > > >> > > > segment.ms > > > >> > > > >>>> to 1, > > > >> > > > >>>> > > > and I > > > >> > > > >>>> > > > > > don't want to break this assumption. > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann > > > >> > > > >>>> > <br...@zendesk.com.invalid> > > > >> > > > >>>> > > > > > wrote: > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the condition > > is > > > >> met > > > >> > > > right? > > > >> > > > >>>> So > > > >> > > > >>>> > I'm > > > >> > > > >>>> > > > > > > struggling to understand the relevance of > > > >> segment.ms > > > >> > > > here. > > > >> > > > >>>> > Maybe an > > > >> > > > >>>> > > > > > > example > > > >> > > > >>>> > > > > > > would help my understanding: > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999 > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1 > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms > > the > > > >> topic > > > >> > > > >>>> should be > > > >> > > > >>>> > > > eligible > > > >> > > > >>>> > > > > > > for compaction when the log compaction thread > > > gets > > > >> > > around > > > >> > > > to > > > >> > > > >>>> > > > evaluating > > > >> > > > >>>> > > > > > the > > > >> > > > >>>> > > > > > > topic. > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > if we have > > > >> > > > >>>> > > > > > > segment.ms=9999999 > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0 > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms > > the > > > >> topic > > > >> > > > >>>> should be > > > >> > > > >>>> > > > eligible > > > >> > > > >>>> > > > > > > for compaction when the log compaction thread > > > gets > > > >> > > around > > > >> > > > to > > > >> > > > >>>> > > > evaluating > > > >> > > > >>>> > > > > > the > > > >> > > > >>>> > > > > > > topic. > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > In both of those cases the change would mean a > > > new > > > >> > > segment > > > >> > > > >>>> is > > > >> > > > >>>> > rolled > > > >> > > > >>>> > > > so > > > >> > > > >>>> > > > > > the > > > >> > > > >>>> > > > > > > new message would be part of the compaction > > > task. 0 > > > >> > and > > > >> > > 1 > > > >> > > > >>>> are > > > >> > > > >>>> > > > practically > > > >> > > > >>>> > > > > > > the same meaning since neither is providing an > > > >> actual > > > >> > > > >>>> guarantee > > > >> > > > >>>> > at > > > >> > > > >>>> > > > such > > > >> > > > >>>> > > > > > low > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it to both > > the > > > >> > > > frequency > > > >> > > > >>>> of > > > >> > > > >>>> > the > > > >> > > > >>>> > > > log > > > >> > > > >>>> > > > > > > cleaner running and the priority of the given > > > topic > > > >> > > being > > > >> > > > >>>> the > > > >> > > > >>>> > highest > > > >> > > > >>>> > > > > > > priority of all topics that are evaluated for > > > >> cleaning > > > >> > > on > > > >> > > > >>>> the > > > >> > > > >>>> > next > > > >> > > > >>>> > > > cycle. > > > >> > > > >>>> > > > > > > You've captured that nuance with careful > > > "skipped" > > > >> > > wording > > > >> > > > >>>> in > > > >> > > > >>>> > the KIP > > > >> > > > >>>> > > > > > > here "controls > > > >> > > > >>>> > > > > > > the max time interval a message/segment can be > > > >> skipped > > > >> > > for > > > >> > > > >>>> log > > > >> > > > >>>> > > > > > compaction". > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And how > > is > > > it > > > >> > > > >>>> relating to > > > >> > > > >>>> > > > > > segment.ms > > > >> > > > >>>> > > > > > > ? > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use > > > >> > > > segment.ms > > > >> > > > >>>> > > > instead?" as > > > >> > > > >>>> > > > > > a > > > >> > > > >>>> > > > > > > kind of third option? > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu < > > > >> > > > >>>> xiongq...@gmail.com> > > > >> > > > >>>> > > > wrote: > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > To make it clear, > > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we > > > >> need to > > > >> > > > come > > > >> > > > >>>> up > > > >> > > > >>>> > with > > > >> > > > >>>> > > > the > > > >> > > > >>>> > > > > > > > meaning of "0". > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but the > > > actual > > > >> > > > >>>> compaction > > > >> > > > >>>> > lag > > > >> > > > >>>> > > > will > > > >> > > > >>>> > > > > > be > > > >> > > > >>>> > > > > > > > segment.ms. > > > >> > > > >>>> > > > > > > > It has longer lag than setting the value to > > be > > > >> half > > > >> > of > > > >> > > > >>>> > segment.ms. > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag. > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > Here are two options. > > > >> > > > >>>> > > > > > > > Option 1: > > > >> > > > >>>> > > > > > > > Keep 0 as disabled > > > >> > > > >>>> > > > > > > > Option 2: > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = > > > >> segment.ms), > > > >> > > and > > > >> > > > >>>> > others. > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann > > > >> > > > >>>> > > > <br...@zendesk.com.invalid> > > > >> > > > >>>> > > > > > > > wrote: > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with these > > > >> settings > > > >> > > for > > > >> > > > >>>> > example: > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes > > > >> > > > >>>> > > > > > > > > retention.bytes > > > >> > > > >>>> > > > > > > > > retention.ms > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > and acks. > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS > > defaults, > > > >> max > > > >> > > > (acks), > > > >> > > > >>>> > etc. I > > > >> > > > >>>> > > > don't > > > >> > > > >>>> > > > > > > see > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those things. > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion > > > >> there > > > >> > is > > > >> > > > >>>> NULL is > > > >> > > > >>>> > used > > > >> > > > >>>> > > > > > where > > > >> > > > >>>> > > > > > > > > there's another setting in the hierarchy > > that > > > >> > would > > > >> > > be > > > >> > > > >>>> used > > > >> > > > >>>> > > > instead. > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann > > < > > > >> > > > >>>> > br...@zendesk.com> > > > >> > > > >>>> > > > > > wrote: > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then > > > we're > > > >> not > > > >> > > > being > > > >> > > > >>>> > > > consistent > > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw > > > out > > > >> -1 > > > >> > > for > > > >> > > > >>>> > > > consideration > > > >> > > > >>>> > > > > > > > > > again :) > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi > > wu > > > < > > > >> > > > >>>> > > > xiongq...@gmail.com> > > > >> > > > >>>> > > > > > > > wrote: > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is > > > merged. > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of " > > > >> segment.ms" > > > >> > > to > > > >> > > > >>>> 1, we > > > >> > > > >>>> > > > cannot > > > >> > > > >>>> > > > > > set > > > >> > > > >>>> > > > > > > " > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0. > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> I also updated the title of this KIP. > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett > > Rann > > > >> > > > >>>> > > > > > <br...@zendesk.com.invalid > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm > > > in > > > >> the > > > >> > > > >>>> minority. > > > >> > > > >>>> > :) > > > >> > > > >>>> > > > Can > > > >> > > > >>>> > > > > > we > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning > > > >> > disabled > > > >> > > > >>>> here: > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > > > >> > > > >>>> > > > > > < > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>> > > > >> > > > >>>> > > > > > > > < > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > > > >> > > > >>>> > > > > > < > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs > > >>> > > > >> > > > >>>> > > > > > > > > >> < > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > > > >> > > > >>>> > > > > > < > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>> > > > >> > > > >>>> > > > > > > > < > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs> > > > >> > > > >>>> > > > > > < > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs > > >>>> > > > ? > > > >> > > > >>>> > > > > > > > > >> > And while there it would be good if > > > >> > segment.ms > > > >> > > > is > > > >> > > > >>>> > > > documented > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some > > > >> > hierarchy > > > >> > > > of > > > >> > > > >>>> > configs > > > >> > > > >>>> > > > for > > > >> > > > >>>> > > > > > > that > > > >> > > > >>>> > > > > > > > > too > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for others > > means > > > >> > > > disabled!) > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi > > > wu > > > >> < > > > >> > > > >>>> > > > xiongq...@gmail.com> > > > >> > > > >>>> > > > > > > > > wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate > > > >> > compaction, > > > >> > > > the > > > >> > > > >>>> > > > compaction > > > >> > > > >>>> > > > > > lag > > > >> > > > >>>> > > > > > > > is > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst > > > >> case. If > > > >> > > > >>>> segment.ms > > > >> > > > >>>> > is > > > >> > > > >>>> > > > 24 > > > >> > > > >>>> > > > > > > > hours, > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker > > > >> > guarantee > > > >> > > > than > > > >> > > > >>>> > setting > > > >> > > > >>>> > > > any > > > >> > > > >>>> > > > > > > > value > > > >> > > > >>>> > > > > > > > > >> less > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of > > > "max > > > >> > > > >>>> compaction > > > >> > > > >>>> > lag", > > > >> > > > >>>> > > > we > > > >> > > > >>>> > > > > > > > cannot > > > >> > > > >>>> > > > > > > > > >> have > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate > > > >> "disable". > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin > > > >> > McCabe < > > > >> > > > >>>> > > > > > cmcc...@apache.org > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, > > Brett > > > >> Rann > > > >> > > > >>>> wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should > > > >> make > > > >> > 0 = > > > >> > > > >>>> > disable, to > > > >> > > > >>>> > > > be > > > >> > > > >>>> > > > > > > > > >> consistent > > > >> > > > >>>> > > > > > > > > >> > > with > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable > > > and > > > >> > when > > > >> > > > >>>> seeing > > > >> > > > >>>> > it > > > >> > > > >>>> > > > in a > > > >> > > > >>>> > > > > > > > config > > > >> > > > >>>> > > > > > > > > >> it's > > > >> > > > >>>> > > > > > > > > >> > > > clear > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0 > > > >> doesn't > > > >> > > have > > > >> > > > >>>> to > > > >> > > > >>>> > mean > > > >> > > > >>>> > > > > > instant, > > > >> > > > >>>> > > > > > > > it > > > >> > > > >>>> > > > > > > > > >> just > > > >> > > > >>>> > > > > > > > > >> > > > means > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't > > > >> think 0 > > > >> > > is > > > >> > > > >>>> > intuitive > > > >> > > > >>>> > > > for > > > >> > > > >>>> > > > > > > > > disabled > > > >> > > > >>>> > > > > > > > > >> and > > > >> > > > >>>> > > > > > > > > >> > > it > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't > > aware > > > >> > > > segment.ms=0 > > > >> > > > >>>> == > > > >> > > > >>>> > > > > > disabled, > > > >> > > > >>>> > > > > > > > > but I > > > >> > > > >>>> > > > > > > > > >> > > think > > > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive. > > > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument for > > > >> keeping > > > >> > > > these > > > >> > > > >>>> two > > > >> > > > >>>> > > > > > > > configurations > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so > > > >> similar. I > > > >> > > > agree > > > >> > > > >>>> that > > > >> > > > >>>> > 0 > > > >> > > > >>>> > > > was an > > > >> > > > >>>> > > > > > > > > >> > unfortunate > > > >> > > > >>>> > > > > > > > > >> > > > choice., > > > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > best, > > > >> > > > >>>> > > > > > > > > >> > > > Colin > > > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM > > > >> Colin > > > >> > > > McCabe > > > >> > > > >>>> < > > > >> > > > >>>> > > > > > > > > cmcc...@apache.org> > > > >> > > > >>>> > > > > > > > > >> > > wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47, > > > >> > xiongqi > > > >> > > wu > > > >> > > > >>>> wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin, > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments. > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 > > PM > > > >> > Colin > > > >> > > > >>>> McCabe < > > > >> > > > >>>> > > > > > > > > >> cmcc...@apache.org> > > > >> > > > >>>> > > > > > > > > >> > > > wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit > > > >> ambiguous. > > > >> > Our > > > >> > > > >>>> > compaction > > > >> > > > >>>> > > > > > > policies > > > >> > > > >>>> > > > > > > > > are > > > >> > > > >>>> > > > > > > > > >> > > already > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It > > > >> seems > > > >> > > like > > > >> > > > >>>> this > > > >> > > > >>>> > > > change is > > > >> > > > >>>> > > > > > > > > focused > > > >> > > > >>>> > > > > > > > > >> > > around > > > >> > > > >>>> > > > > > > > > >> > > > > > adding > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms > > ." > > > >> > Perhaps > > > >> > > > >>>> the KIP > > > >> > > > >>>> > > > title > > > >> > > > >>>> > > > > > > should > > > >> > > > >>>> > > > > > > > > be > > > >> > > > >>>> > > > > > > > > >> > > > something > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum > > compaction > > > >> lag > > > >> > > > time"? > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will > > > >> change > > > >> > the > > > >> > > > >>>> title. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is > > forced > > > >> to > > > >> > > roll > > > >> > > > >>>> when > > > >> > > > >>>> > > > either " > > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms" > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" ( > > > >> log.roll.ms > > > >> > and > > > >> > > > >>>> > > > log.roll.hours) > > > >> > > > >>>> > > > > > > has > > > >> > > > >>>> > > > > > > > > >> > reached. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the > > > max.compaction.lag.ms > > > >> is > > > >> > > low, > > > >> > > > >>>> it > > > >> > > > >>>> > seems > > > >> > > > >>>> > > > like > > > >> > > > >>>> > > > > > > > > segments > > > >> > > > >>>> > > > > > > > > >> > will > > > >> > > > >>>> > > > > > > > > >> > > > be > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. > > This > > > >> can > > > >> > > be a > > > >> > > > >>>> > source of > > > >> > > > >>>> > > > > > > problems > > > >> > > > >>>> > > > > > > > > in > > > >> > > > >>>> > > > > > > > > >> the > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many > > > different > > > >> > small > > > >> > > > log > > > >> > > > >>>> > segments > > > >> > > > >>>> > > > > > > > consumes > > > >> > > > >>>> > > > > > > > > a > > > >> > > > >>>> > > > > > > > > >> > huge > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources. > > > Therefore, > > > >> I > > > >> > > would > > > >> > > > >>>> > suggest > > > >> > > > >>>> > > > > > adding a > > > >> > > > >>>> > > > > > > > > >> > > broker-level > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows > > > us > > > >> to > > > >> > > set > > > >> > > > a > > > >> > > > >>>> > minimum > > > >> > > > >>>> > > > value > > > >> > > > >>>> > > > > > > for > > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If > > > we > > > >> let > > > >> > > > >>>> users set > > > >> > > > >>>> > it > > > >> > > > >>>> > > > on a > > > >> > > > >>>> > > > > > > > > >> per-topic > > > >> > > > >>>> > > > > > > > > >> > > > basis, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value > > of > > > >> 1 ms > > > >> > > or > > > >> > > > >>>> > something, > > > >> > > > >>>> > > > and > > > >> > > > >>>> > > > > > > > cause > > > >> > > > >>>> > > > > > > > > >> > chaos. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to > > > >> > > > segment.ms > > > >> > > > >>>> as > > > >> > > > >>>> > well. > > > >> > > > >>>> > > > > > Today > > > >> > > > >>>> > > > > > > > > users > > > >> > > > >>>> > > > > > > > > >> can > > > >> > > > >>>> > > > > > > > > >> > > > set " > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low > > > >> value, > > > >> > and > > > >> > > > >>>> cause a > > > >> > > > >>>> > > > frequent > > > >> > > > >>>> > > > > > > > > >> rolling of > > > >> > > > >>>> > > > > > > > > >> > > > active > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an > > existing > > > >> > > problem > > > >> > > > >>>> with > > > >> > > > >>>> > > > > > segment.ms. > > > >> > > > >>>> > > > > > > > > >> However, > > > >> > > > >>>> > > > > > > > > >> > > > that > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't > > > fix > > > >> > it. > > > >> > > As > > > >> > > > >>>> you > > > >> > > > >>>> > noted, > > > >> > > > >>>> > > > > > there > > > >> > > > >>>> > > > > > > > > will > > > >> > > > >>>> > > > > > > > > >> be > > > >> > > > >>>> > > > > > > > > >> > > more > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level > > > >> > retention > > > >> > > > >>>> > settings as > > > >> > > > >>>> > > > a > > > >> > > > >>>> > > > > > > result > > > >> > > > >>>> > > > > > > > > of > > > >> > > > >>>> > > > > > > > > >> > GDPR. > > > >> > > > >>>> > > > > > > > > >> > > > It > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing > > > >> > problems > > > >> > > > >>>> will > > > >> > > > >>>> > cause > > > >> > > > >>>> > > > more > > > >> > > > >>>> > > > > > > > > trouble. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively > > > >> > > straightforward > > > >> > > > >>>> here -- > > > >> > > > >>>> > > > add a > > > >> > > > >>>> > > > > > > > > >> broker-level > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that > > > overrides > > > >> > > > >>>> per-topic > > > >> > > > >>>> > > > minimums. > > > >> > > > >>>> > > > > > We > > > >> > > > >>>> > > > > > > > can > > > >> > > > >>>> > > > > > > > > >> also > > > >> > > > >>>> > > > > > > > > >> > > fail > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message > > > when > > > >> > > someone > > > >> > > > >>>> > attempts > > > >> > > > >>>> > > > to > > > >> > > > >>>> > > > > > set > > > >> > > > >>>> > > > > > > an > > > >> > > > >>>> > > > > > > > > >> > invalid > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum > > of " > > > >> > > > >>>> > > > max.compaction.lag.ms" > > > >> > > > >>>> > > > > > > > should > > > >> > > > >>>> > > > > > > > > >> be > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of " > > > >> > segment.ms > > > >> > > ". > > > >> > > > >>>> Since > > > >> > > > >>>> > > > today the > > > >> > > > >>>> > > > > > > > > >> minimum of > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, " > > max.compaction.lag.ms" > > > >> also > > > >> > > > >>>> starts > > > >> > > > >>>> > with > > > >> > > > >>>> > > > 1. "0" > > > >> > > > >>>> > > > > > > > means > > > >> > > > >>>> > > > > > > > > >> > > > disable. I > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but > > it > > > is > > > >> > hard > > > >> > > > to > > > >> > > > >>>> > define > > > >> > > > >>>> > > > the > > > >> > > > >>>> > > > > > > > meaning > > > >> > > > >>>> > > > > > > > > >> of 0 > > > >> > > > >>>> > > > > > > > > >> > > > because > > > >> > > > >>>> > > > > > > > > >> > > > > > we > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active > > > >> segment > > > >> > > > >>>> > immediately. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should > > > >> make > > > >> > 0 = > > > >> > > > >>>> > disable, to > > > >> > > > >>>> > > > be > > > >> > > > >>>> > > > > > > > > >> consistent > > > >> > > > >>>> > > > > > > > > >> > > with > > > >> > > > >>>> > > > > > > > > >> > > > the > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best, > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an > > alternative > > > >> > > > >>>> configuration > > > >> > > > >>>> > is to > > > >> > > > >>>> > > > use > > > >> > > > >>>> > > > > > -1 > > > >> > > > >>>> > > > > > > > as > > > >> > > > >>>> > > > > > > > > >> > > > "disabled" > > > >> > > > >>>> > > > > > > > > >> > > > > > and > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0 > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate > > compaction". > > > >> > > Because > > > >> > > > >>>> > compaction > > > >> > > > >>>> > > > lag > > > >> > > > >>>> > > > > > is > > > >> > > > >>>> > > > > > > > > still > > > >> > > > >>>> > > > > > > > > >> > > > determined > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on > > > min.compaction.lag > > > >> and > > > >> > > how > > > >> > > > >>>> long > > > >> > > > >>>> > to > > > >> > > > >>>> > > > roll > > > >> > > > >>>> > > > > > an > > > >> > > > >>>> > > > > > > > > active > > > >> > > > >>>> > > > > > > > > >> > > > segment, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for > > compaction > > > is > > > >> > > > >>>> undetermined > > > >> > > > >>>> > if > > > >> > > > >>>> > > > we > > > >> > > > >>>> > > > > > use > > > >> > > > >>>> > > > > > > > "0". > > > >> > > > >>>> > > > > > > > > >> On > > > >> > > > >>>> > > > > > > > > >> > the > > > >> > > > >>>> > > > > > > > > >> > > > other > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio" > > > >> > > > >>>> > > > > > > > to > > > >> > > > >>>> > > > > > > > > >> > achieve > > > >> > > > >>>> > > > > > > > > >> > > > the > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose > > > "0" > > > >> as > > > >> > > > >>>> "disabled". > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be > > the > > > >> > invalid > > > >> > > > >>>> setting. > > > >> > > > >>>> > > > > > Treating > > > >> > > > >>>> > > > > > > 0 > > > >> > > > >>>> > > > > > > > > >> > > differently > > > >> > > > >>>> > > > > > > > > >> > > > > > than > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous > > > >> comment, I > > > >> > > am > > > >> > > > >>>> not > > > >> > > > >>>> > > > strongly > > > >> > > > >>>> > > > > > > > against, > > > >> > > > >>>> > > > > > > > > >> but > > > >> > > > >>>> > > > > > > > > >> > 0 > > > >> > > > >>>> > > > > > > > > >> > > is > > > >> > > > >>>> > > > > > > > > >> > > > > > not a > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my > > > >> option. > > > >> > So > > > >> > > I > > > >> > > > >>>> use > > > >> > > > >>>> > "0" as > > > >> > > > >>>> > > > > > > disabled > > > >> > > > >>>> > > > > > > > > >> state. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best, > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at > > > 15:04, > > > >> > > > xiongqi > > > >> > > > >>>> wu > > > >> > > > >>>> > wrote: > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP. > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP: > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>> > > > >> > > > >>>> > > > > > > > > >> < > > > >> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>>> > > > >> > > > >>>> > > > > > > > > >> > > < > > > >> > > > >>>> > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>> > > > >> > > > >>>> > > > > > > > > >> < > > > >> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>>>> > > > >> > > > >>>> > > > > > > > > >> > > > > > < > > > >> > > > >>>> > > > > > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>> > > > >> > > > >>>> > > > > > > > > >> < > > > >> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>>> > > > >> > > > >>>> > > > > > > > > >> > > < > > > >> > > > >>>> > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> >>> > > > >> > > > >>>> > > > > > > > > >> < > > > >> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >> > > > >>>> > > > > > > > < > > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > >> > > > >>>> > > > > > < > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > > > >>>> > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > >> > >>>>>> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > %3A+Time-based+log+compaction+policy > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation: > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > > > >> > > > >>>> > > > > > > > > >> < > > > https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>> > > > >> > > > >>>> > > > > > > > > >> > > < > > > >> https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > > > >> > > > >>>> > > > > > > > > >> < > > > https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>> > > > >> > > > >>>> > > > > > > > > >> > > > > > < > > > >> > > > https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > > > >> > > > >>>> > > > > > > > > >> < > > > https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>> > > > >> > > > >>>> > > > > > > > > >> > > < > > > >> https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>> > > > >> > > > >>>> > > > > > > > > >> < > > > https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611 > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > -- > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000 > > > >> > > Australia > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017 > > > >> > > > >>>> > > > > > > > > >> > > > > >> > > > >>>> > > > > > > > > >> > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > -- > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Brett Rann > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 > > > >> Australia > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017 > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > -- > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > Brett Rann > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 > > > >> Australia > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017 > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > >> > > > >>>> > > > > >> > > > >>>> > > > >> > > > >>> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > >
