Thanks Colin, Dong, and Joel for voting. I have masked this KIP as accepted.
The pull request is up: https://github.com/apache/kafka/pull/6009 Xiongqi (Wesley) Wu On Thu, Dec 6, 2018 at 10:47 AM Joel Koshy <jjkosh...@gmail.com> wrote: > +1 on the updated KIP. > > On Wed, Dec 5, 2018 at 11:56 AM Dong Lin <lindon...@gmail.com> wrote: > > > Thanks for the update. +1 (binding) > > > > On Wed, Dec 5, 2018 at 8:19 AM Colin McCabe <cmcc...@apache.org> wrote: > > > > > Thanks, Xiongqi Wu. +1 (binding) > > > > > > regards, > > > Colin > > > > > > > > > On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote: > > > > Colin, > > > > > > > > Thanks for comments. > > > > Out of ordered message timestamp is a very good point. > > > > We can combine max.compaction.lag.ms with > > > > log.message.timestamp.difference.max.ms to achieve what we want in > an > > > > environment that message timestamp can be shifted a lot. > > > > > > > > There are similar discussions regarding log.retention.ms and > > > > log.message.timestamp.difference.max.ms in KAFKA-4340. > > > > > > > > I agree that we can always use first message timestamp not the > > > maxTimestamp > > > > of the previous log segment to make it simple. > > > > > > > > I have updated the KIP. > > > > > > > > Xiongqi (wesley) Wu > > > > > > > > > > > > On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cmcc...@apache.org> > > wrote: > > > > > > > > > Hi Xiongqi, > > > > > > > > > > Thinking about this a little bit more, it seems like we don't have > > any > > > > > guarantees just by looking at the timestamp of the first message > in a > > > log > > > > > segment. Similarly, we don't have any guarantees just by looking > at > > > the > > > > > maxTimestamp of the previous log segment. Old data could appear > > > anywhere-- > > > > > you could put data that was years old in the middle of a segment > from > > > 2018. > > > > > > > > > > However, if log.message.timestamp.difference.max.ms is set, then > we > > > can > > > > > make some actual guarantees that old data gets purged-- which is > what > > > the > > > > > GDPR requires, of course. > > > > > > > > > > Overall, maybe we can make KIP-354 simpler by just always looking > at > > > the > > > > > timestamp of the first log message. I don't think looking at the > > > > > maxTimestamp of the previous segment is any more accurate. Aside > > from > > > > > that, looks good, since we can get what we need with the > combination > > of > > > > > this and log.message.timestamp.difference.max.ms. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote: > > > > > > Thanks for binding and non-binding votes. > > > > > > Can I get one more binding vote? > > > > > > > > > > > > Thanks in advance! > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > > > > > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <m...@frmr.me> > wrote: > > > > > > > > > > > > > I'm a +1 (non-binding) — This looks like it would have saved > us a > > > lot > > > > > of > > > > > > > pain in an issue we had to debug recently. I can't go into > > > details, but > > > > > > > figuring out how to achieve this effect gave me quite a > headache. > > > :) > > > > > > > > > > > > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu < > xiongq...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Can I have one more vote on this KIP? > > > > > > > > Any comment is appreciated. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag > > > > > > > > > > > > > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu < > xiongq...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Dong. > > > > > > > > > I have updated the KIP. > > > > > > > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin < > lindon...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding) > > > > > > > > >> > > > > > > > > >> One minor comment: it may be a bit better to clarify in > the > > > public > > > > > > > > >> interface section that the value of the newly added metric > > is > > > > > > > determined > > > > > > > > >> based by applying that formula across all compactable > > > segments. > > > > > For > > > > > > > > >> example: > > > > > > > > >> > > > > > > > > >> The maximum value of Math.max(now - > > > > > > > > >> earliest_timestamp_in_ms_of_uncompacted_segment - > > > > > > > max.compaction.lag.ms > > > > > > > > , > > > > > > > > >> 0)/1000 across all compactable partitions, where the > > > > > > > > >> max.compaction.lag.ms > > > > > > > > >> can be overridden on per-topic basis. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu < > > > xiongq...@gmail.com> > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> > Thanks Joel. > > > > > > > > >> > Tracking the delay at second granularity makes sense > > > > > > > > >> > I have updated KIP. > > > > > > > > >> > > > > > > > > > >> > Xiongqi (Wesley) Wu > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy < > > > jjkosh...@gmail.com> > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > >> > > +1 with one suggestion on the proposed metric. You > > should > > > > > probably > > > > > > > > >> > include > > > > > > > > >> > > the unit. So for e.g., max-compaction-delay-secs. > > > > > > > > >> > > > > > > > > > > >> > > Joel > > > > > > > > >> > > > > > > > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu < > > > > > xiongq...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > bump > > > > > > > > >> > > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu < > > > > > xiongq...@gmail.com > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and > > Xiaohe > > > for > > > > > > > > >> feedback. > > > > > > > > >> > > > > Can I have more feedback or VOTE on this KIP? > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu < > > > > > > > > xiongq...@gmail.com> > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > >> Any other votes or comments? > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Xiongqi (Wesley) Wu > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu < > > > > > > > > xiongq...@gmail.com > > > > > > > > >> > > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >> > > > > > > > > >> > > > >>> Yes, more votes and code review. > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann > > > > > > > > >> > > <br...@zendesk.com.invalid > > > > > > > > >> > > > > > > > > > > > > >> > > > >>> wrote: > > > > > > > > >> > > > >>> > > > > > > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the > KIP. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> Where do we go from here? More votes? > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe < > > > > > > > > >> cmcc...@apache.org> > > > > > > > > >> > > > >>>> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu > > wrote: > > > > > > > > >> > > > >>>> > > Thank you for comments. I will use '0' for > > now. > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > If we create topics through admin client in > > the > > > > > future, > > > > > > > > we > > > > > > > > >> > might > > > > > > > > >> > > > >>>> perform > > > > > > > > >> > > > >>>> > > some useful checks. (but the assumption is > > all > > > > > brokers > > > > > > > in > > > > > > > > >> the > > > > > > > > >> > > same > > > > > > > > >> > > > >>>> > cluster > > > > > > > > >> > > > >>>> > > have the same default configurations value, > > > > > > > otherwise,it > > > > > > > > >> might > > > > > > > > >> > > > >>>> still be > > > > > > > > >> > > > >>>> > > tricky to do such cross validation check.) > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > This isn't something that we might do in the > > > future-- > > > > > > > this > > > > > > > > is > > > > > > > > >> > > > >>>> something we > > > > > > > > >> > > > >>>> > are doing now. We already have Create Topic > > > policies > > > > > > > which > > > > > > > > >> are > > > > > > > > >> > > > >>>> enforced by > > > > > > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for > > > details. > > > > > This > > > > > > > is > > > > > > > > >> one > > > > > > > > >> > of > > > > > > > > >> > > > the > > > > > > > > >> > > > >>>> > motivations for getting rid of direct ZK > > access-- > > > > > making > > > > > > > > sure > > > > > > > > >> > that > > > > > > > > >> > > > >>>> these > > > > > > > > >> > > > >>>> > policies are applied. > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > I agree that having different configurations > on > > > > > different > > > > > > > > >> > brokers > > > > > > > > >> > > > can > > > > > > > > >> > > > >>>> be > > > > > > > > >> > > > >>>> > confusing and frustrating . That's why more > > > > > > > configurations > > > > > > > > >> are > > > > > > > > >> > > being > > > > > > > > >> > > > >>>> made > > > > > > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations > > are > > > > > stored > > > > > > > > >> > centrally > > > > > > > > >> > > > in > > > > > > > > >> > > > >>>> ZK, > > > > > > > > >> > > > >>>> > so they are the same on all brokers (modulo > > > > > propagation > > > > > > > > >> delays). > > > > > > > > >> > > In > > > > > > > > >> > > > >>>> any > > > > > > > > >> > > > >>>> > case, this is a general issue, not specific > to > > > > > "create > > > > > > > > >> topics". > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > cheers, > > > > > > > > >> > > > >>>> > Colin > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin > > McCabe < > > > > > > > > >> > > cmcc...@apache.org > > > > > > > > >> > > > > > > > > > > > > >> > > > >>>> > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > >> > > > >>>> > > > I don't have a strong opinion. But I > think > > we > > > > > should > > > > > > > > >> > probably > > > > > > > > >> > > be > > > > > > > > >> > > > >>>> > > > consistent with how segment.ms works, > and > > > just > > > > > use > > > > > > > 0. > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > best, > > > > > > > > >> > > > >>>> > > > Colin > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann > > > wrote: > > > > > > > > >> > > > >>>> > > > > OK thanks for that clarification. I see > > why > > > > > you're > > > > > > > > >> > > > uncomfortable > > > > > > > > >> > > > >>>> > with 0 > > > > > > > > >> > > > >>>> > > > now. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer > > > > > consistency in > > > > > > > > >> > > > >>>> configuration > > > > > > > > >> > > > >>>> > > > options. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 > and > > 1 > > > > > > > similarly > > > > > > > > in > > > > > > > > >> > that > > > > > > > > >> > > > >>>> > scenario, > > > > > > > > >> > > > >>>> > > > > because it favours the person thinking > > > about > > > > > > > setting > > > > > > > > >> the > > > > > > > > >> > > > >>>> > configurations, > > > > > > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms > > edge > > > case > > > > > > > > >> especially > > > > > > > > >> > > > when > > > > > > > > >> > > > >>>> the > > > > > > > > >> > > > >>>> > > > context > > > > > > > > >> > > > >>>> > > > > is the true minimum is tied to the log > > > cleaner > > > > > > > > cadence. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" > because > > > there > > > > > is > > > > > > > > some > > > > > > > > >> > > > >>>> uniquness in > > > > > > > > >> > > > >>>> > > > > segment.ms not being able to be set to > > 0, > > > > > reduces > > > > > > > > >> > > > configuration > > > > > > > > >> > > > >>>> > > > consistency > > > > > > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an > > edge > > > case > > > > > > > that > > > > > > > > >> > nobody > > > > > > > > >> > > > >>>> would > > > > > > > > >> > > > >>>> > ever > > > > > > > > >> > > > >>>> > > > > notice. For someone to understand why > > > > > everywhere > > > > > > > else > > > > > > > > >> -1 > > > > > > > > >> > is > > > > > > > > >> > > > >>>> used to > > > > > > > > >> > > > >>>> > > > > disable, but here 0 is used, they would > > > need to > > > > > > > learn > > > > > > > > >> > about > > > > > > > > >> > > > >>>> > segment.ms > > > > > > > > >> > > > >>>> > > > > having a 1ms minimum and then after > > > learning > > > > > would > > > > > > > > >> think > > > > > > > > >> > > "who > > > > > > > > >> > > > >>>> cares > > > > > > > > >> > > > >>>> > about > > > > > > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway > :) > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to > > majority. > > > > > Curious > > > > > > > > >> which > > > > > > > > >> > > way > > > > > > > > >> > > > >>>> Colin > > > > > > > > >> > > > >>>> > falls > > > > > > > > >> > > > >>>> > > > > now. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > Don't want to spend more time on this > > > though, > > > > > It's > > > > > > > > well > > > > > > > > >> > into > > > > > > > > >> > > > >>>> > > > bikeshedding > > > > > > > > >> > > > >>>> > > > > territory now. :) > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi > > wu < > > > > > > > > >> > > > xiongq...@gmail.com> > > > > > > > > >> > > > >>>> > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of > > > > > segment.ms > > > > > > > > >> (which > > > > > > > > >> > is > > > > > > > > >> > > > >>>> 1ms) to > > > > > > > > >> > > > >>>> > > > force > > > > > > > > >> > > > >>>> > > > > > roll an active segment. > > > > > > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" > > any > > > > > value > > > > > > > > > 0, > > > > > > > > >> the > > > > > > > > >> > > > >>>> minimum of > > > > > > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms > > > will be > > > > > > > used > > > > > > > > to > > > > > > > > >> > seal > > > > > > > > >> > > > an > > > > > > > > >> > > > >>>> > active > > > > > > > > >> > > > >>>> > > > > > segment. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to > 0, > > > the > > > > > > > current > > > > > > > > >> > > > >>>> implementation > > > > > > > > >> > > > >>>> > will > > > > > > > > >> > > > >>>> > > > > > treat it as disabled. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat > > > > > > > > >> max.compaction.lag=0 > > > > > > > > >> > the > > > > > > > > >> > > > >>>> same as > > > > > > > > >> > > > >>>> > > > > > max.compaction.lag=1. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > There might be a reason why we set > the > > > > > minimum of > > > > > > > > >> > > > segment.ms > > > > > > > > >> > > > >>>> to 1, > > > > > > > > >> > > > >>>> > > > and I > > > > > > > > >> > > > >>>> > > > > > don't want to break this assumption. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett > > Rann > > > > > > > > >> > > > >>>> > <br...@zendesk.com.invalid> > > > > > > > > >> > > > >>>> > > > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the > > > > > condition > > > > > > > is > > > > > > > > >> met > > > > > > > > >> > > > right? > > > > > > > > >> > > > >>>> So > > > > > > > > >> > > > >>>> > I'm > > > > > > > > >> > > > >>>> > > > > > > struggling to understand the > > relevance > > > of > > > > > > > > >> segment.ms > > > > > > > > >> > > > here. > > > > > > > > >> > > > >>>> > Maybe an > > > > > > > > >> > > > >>>> > > > > > > example > > > > > > > > >> > > > >>>> > > > > > > would help my understanding: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999 > > > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > > > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1 > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, > > > after > > > > > 1ms > > > > > > > the > > > > > > > > >> topic > > > > > > > > >> > > > >>>> should be > > > > > > > > >> > > > >>>> > > > eligible > > > > > > > > >> > > > >>>> > > > > > > for compaction when the log > > compaction > > > > > thread > > > > > > > > gets > > > > > > > > >> > > around > > > > > > > > >> > > > to > > > > > > > > >> > > > >>>> > > > evaluating > > > > > > > > >> > > > >>>> > > > > > the > > > > > > > > >> > > > >>>> > > > > > > topic. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > if we have > > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999 > > > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1* > > > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0 > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, > > > after > > > > > 0ms > > > > > > > the > > > > > > > > >> topic > > > > > > > > >> > > > >>>> should be > > > > > > > > >> > > > >>>> > > > eligible > > > > > > > > >> > > > >>>> > > > > > > for compaction when the log > > compaction > > > > > thread > > > > > > > > gets > > > > > > > > >> > > around > > > > > > > > >> > > > to > > > > > > > > >> > > > >>>> > > > evaluating > > > > > > > > >> > > > >>>> > > > > > the > > > > > > > > >> > > > >>>> > > > > > > topic. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > In both of those cases the change > > would > > > > > mean a > > > > > > > > new > > > > > > > > >> > > segment > > > > > > > > >> > > > >>>> is > > > > > > > > >> > > > >>>> > rolled > > > > > > > > >> > > > >>>> > > > so > > > > > > > > >> > > > >>>> > > > > > the > > > > > > > > >> > > > >>>> > > > > > > new message would be part of the > > > compaction > > > > > > > > task. 0 > > > > > > > > >> > and > > > > > > > > >> > > 1 > > > > > > > > >> > > > >>>> are > > > > > > > > >> > > > >>>> > > > practically > > > > > > > > >> > > > >>>> > > > > > > the same meaning since neither is > > > > > providing an > > > > > > > > >> actual > > > > > > > > >> > > > >>>> guarantee > > > > > > > > >> > > > >>>> > at > > > > > > > > >> > > > >>>> > > > such > > > > > > > > >> > > > >>>> > > > > > low > > > > > > > > >> > > > >>>> > > > > > > MS settings, but effectively tying > it > > > to > > > > > both > > > > > > > the > > > > > > > > >> > > > frequency > > > > > > > > >> > > > >>>> of > > > > > > > > >> > > > >>>> > the > > > > > > > > >> > > > >>>> > > > log > > > > > > > > >> > > > >>>> > > > > > > cleaner running and the priority of > > the > > > > > given > > > > > > > > topic > > > > > > > > >> > > being > > > > > > > > >> > > > >>>> the > > > > > > > > >> > > > >>>> > highest > > > > > > > > >> > > > >>>> > > > > > > priority of all topics that are > > > evaluated > > > > > for > > > > > > > > >> cleaning > > > > > > > > >> > > on > > > > > > > > >> > > > >>>> the > > > > > > > > >> > > > >>>> > next > > > > > > > > >> > > > >>>> > > > cycle. > > > > > > > > >> > > > >>>> > > > > > > You've captured that nuance with > > > careful > > > > > > > > "skipped" > > > > > > > > >> > > wording > > > > > > > > >> > > > >>>> in > > > > > > > > >> > > > >>>> > the KIP > > > > > > > > >> > > > >>>> > > > > > > here "controls > > > > > > > > >> > > > >>>> > > > > > > the max time interval a > > message/segment > > > > > can be > > > > > > > > >> skipped > > > > > > > > >> > > for > > > > > > > > >> > > > >>>> log > > > > > > > > >> > > > >>>> > > > > > compaction". > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > How is 0 different to 1, > practically? > > > And > > > > > how > > > > > > > is > > > > > > > > it > > > > > > > > >> > > > >>>> relating to > > > > > > > > >> > > > >>>> > > > > > segment.ms > > > > > > > > >> > > > >>>> > > > > > > ? > > > > > > > > >> > > > >>>> > > > > > > Is it that you're proposing to > have 0 > > > mean > > > > > "use > > > > > > > > >> > > > segment.ms > > > > > > > > >> > > > >>>> > > > instead?" as > > > > > > > > >> > > > >>>> > > > > > a > > > > > > > > >> > > > >>>> > > > > > > kind of third option? > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM > > > xiongqi wu > > > > > < > > > > > > > > >> > > > >>>> xiongq...@gmail.com> > > > > > > > > >> > > > >>>> > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > To make it clear, > > > > > > > > >> > > > >>>> > > > > > > > I don't against using -1 as > > disabled, > > > > > but we > > > > > > > > >> need to > > > > > > > > >> > > > come > > > > > > > > >> > > > >>>> up > > > > > > > > >> > > > >>>> > with > > > > > > > > >> > > > >>>> > > > the > > > > > > > > >> > > > >>>> > > > > > > > meaning of "0". > > > > > > > > >> > > > >>>> > > > > > > > If "0" means immediate > compaction, > > > but > > > > > the > > > > > > > > actual > > > > > > > > >> > > > >>>> compaction > > > > > > > > >> > > > >>>> > lag > > > > > > > > >> > > > >>>> > > > will > > > > > > > > >> > > > >>>> > > > > > be > > > > > > > > >> > > > >>>> > > > > > > > segment.ms. > > > > > > > > >> > > > >>>> > > > > > > > It has longer lag than setting > the > > > value > > > > > to > > > > > > > be > > > > > > > > >> half > > > > > > > > >> > of > > > > > > > > >> > > > >>>> > segment.ms. > > > > > > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max > > > compaction > > > > > lag. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > Here are two options. > > > > > > > > >> > > > >>>> > > > > > > > Option 1: > > > > > > > > >> > > > >>>> > > > > > > > Keep 0 as disabled > > > > > > > > >> > > > >>>> > > > > > > > Option 2: > > > > > > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction > > lag > > > = > > > > > > > > >> segment.ms), > > > > > > > > >> > > and > > > > > > > > >> > > > >>>> > others. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM > > Brett > > > Rann > > > > > > > > >> > > > >>>> > > > <br...@zendesk.com.invalid> > > > > > > > > >> > > > >>>> > > > > > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" > > with > > > > > these > > > > > > > > >> settings > > > > > > > > >> > > for > > > > > > > > >> > > > >>>> > example: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes > > > > > > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes > > > > > > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes > > > > > > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes > > > > > > > > >> > > > >>>> > > > > > > > > retention.bytes > > > > > > > > >> > > > >>>> > > > > > > > > retention.ms > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > and acks. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use > > OS > > > > > > > defaults, > > > > > > > > >> max > > > > > > > > >> > > > (acks), > > > > > > > > >> > > > >>>> > etc. I > > > > > > > > >> > > > >>>> > > > don't > > > > > > > > >> > > > >>>> > > > > > > see > > > > > > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning > > those > > > > > things. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it > seems > > > > > convetion > > > > > > > > >> there > > > > > > > > >> > is > > > > > > > > >> > > > >>>> NULL is > > > > > > > > >> > > > >>>> > used > > > > > > > > >> > > > >>>> > > > > > where > > > > > > > > >> > > > >>>> > > > > > > > > there's another setting in the > > > > > hierarchy > > > > > > > that > > > > > > > > >> > would > > > > > > > > >> > > be > > > > > > > > >> > > > >>>> used > > > > > > > > >> > > > >>>> > > > instead. > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM > > > Brett > > > > > Rann > > > > > > > < > > > > > > > > >> > > > >>>> > br...@zendesk.com> > > > > > > > > >> > > > >>>> > > > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set > to > > 0, > > > > > then > > > > > > > > we're > > > > > > > > >> not > > > > > > > > >> > > > being > > > > > > > > >> > > > >>>> > > > consistent > > > > > > > > >> > > > >>>> > > > > > > > > > by using 0 for this new > > setting? > > > I > > > > > throw > > > > > > > > out > > > > > > > > >> -1 > > > > > > > > >> > > for > > > > > > > > >> > > > >>>> > > > consideration > > > > > > > > >> > > > >>>> > > > > > > > > > again :) > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 > AM > > > > > xiongqi > > > > > > > wu > > > > > > > > < > > > > > > > > >> > > > >>>> > > > xiongq...@gmail.com> > > > > > > > > >> > > > >>>> > > > > > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document > after > > > PR is > > > > > > > > merged. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the > minimum > > > of " > > > > > > > > >> segment.ms" > > > > > > > > >> > > to > > > > > > > > >> > > > >>>> 1, we > > > > > > > > >> > > > >>>> > > > cannot > > > > > > > > >> > > > >>>> > > > > > set > > > > > > > > >> > > > >>>> > > > > > > " > > > > > > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> I also updated the title of > > this > > > > > KIP. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 > PM > > > Brett > > > > > > > Rann > > > > > > > > >> > > > >>>> > > > > > <br...@zendesk.com.invalid > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on > -1 > > > > > since i'm > > > > > > > > in > > > > > > > > >> the > > > > > > > > >> > > > >>>> minority. > > > > > > > > >> > > > >>>> > :) > > > > > > > > >> > > > >>>> > > > Can > > > > > > > > >> > > > >>>> > > > > > we > > > > > > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets > documented > > as > > > > > meaning > > > > > > > > >> > disabled > > > > > > > > >> > > > >>>> here: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs> > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs> > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs> > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs> > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > > >> > > > >>>> > < > > > > > https://kafka.apache.org/documentation/#brokerconfigs > > > > > > > >>>> > > > > > > > > ? > > > > > > > > >> > > > >>>> > > > > > > > > >> > And while there it would > be > > > good > > > > > if > > > > > > > > >> > segment.ms > > > > > > > > >> > > > is > > > > > > > > >> > > > >>>> > > > documented > > > > > > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. > > > (there's > > > > > some > > > > > > > > >> > hierarchy > > > > > > > > >> > > > of > > > > > > > > >> > > > >>>> > configs > > > > > > > > >> > > > >>>> > > > for > > > > > > > > >> > > > >>>> > > > > > > that > > > > > > > > >> > > > >>>> > > > > > > > > too > > > > > > > > >> > > > >>>> > > > > > > > > >> > if its not set and null > for > > > others > > > > > > > means > > > > > > > > >> > > > disabled!) > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at > 4:44 > > AM > > > > > xiongqi > > > > > > > > wu > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > xiongq...@gmail.com> > > > > > > > > >> > > > >>>> > > > > > > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate > > > > > immediate > > > > > > > > >> > compaction, > > > > > > > > >> > > > the > > > > > > > > >> > > > >>>> > > > compaction > > > > > > > > >> > > > >>>> > > > > > lag > > > > > > > > >> > > > >>>> > > > > > > > is > > > > > > > > >> > > > >>>> > > > > > > > > >> > > determined by > segment.ms > > in > > > > > worst > > > > > > > > >> case. If > > > > > > > > >> > > > >>>> segment.ms > > > > > > > > >> > > > >>>> > is > > > > > > > > >> > > > >>>> > > > 24 > > > > > > > > >> > > > >>>> > > > > > > > hours, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" > is > > a > > > > > weaker > > > > > > > > >> > guarantee > > > > > > > > >> > > > than > > > > > > > > >> > > > >>>> > setting > > > > > > > > >> > > > >>>> > > > any > > > > > > > > >> > > > >>>> > > > > > > > value > > > > > > > > >> > > > >>>> > > > > > > > > >> less > > > > > > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the > > > > > definition of > > > > > > > > "max > > > > > > > > >> > > > >>>> compaction > > > > > > > > >> > > > >>>> > lag", > > > > > > > > >> > > > >>>> > > > we > > > > > > > > >> > > > >>>> > > > > > > > cannot > > > > > > > > >> > > > >>>> > > > > > > > > >> have > > > > > > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to > > > indicate > > > > > > > > >> "disable". > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at > > 8:34 > > > AM > > > > > Colin > > > > > > > > >> > McCabe < > > > > > > > > >> > > > >>>> > > > > > cmcc...@apache.org > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, > at > > > 22:11, > > > > > > > Brett > > > > > > > > >> Rann > > > > > > > > >> > > > >>>> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair > point. > > > We > > > > > should > > > > > > > > >> make > > > > > > > > >> > 0 = > > > > > > > > >> > > > >>>> > disable, to > > > > > > > > >> > > > >>>> > > > be > > > > > > > > >> > > > >>>> > > > > > > > > >> consistent > > > > > > > > >> > > > >>>> > > > > > > > > >> > > with > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere > > for > > > > > disable > > > > > > > > and > > > > > > > > >> > when > > > > > > > > >> > > > >>>> seeing > > > > > > > > >> > > > >>>> > it > > > > > > > > >> > > > >>>> > > > in a > > > > > > > > >> > > > >>>> > > > > > > > config > > > > > > > > >> > > > >>>> > > > > > > > > >> it's > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > clear > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special > > > > > meaning. 0 > > > > > > > > >> doesn't > > > > > > > > >> > > have > > > > > > > > >> > > > >>>> to > > > > > > > > >> > > > >>>> > mean > > > > > > > > >> > > > >>>> > > > > > instant, > > > > > > > > >> > > > >>>> > > > > > > > it > > > > > > > > >> > > > >>>> > > > > > > > > >> just > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > means > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as > > possible. > > > I > > > > > don't > > > > > > > > >> think 0 > > > > > > > > >> > > is > > > > > > > > >> > > > >>>> > intuitive > > > > > > > > >> > > > >>>> > > > for > > > > > > > > >> > > > >>>> > > > > > > > > disabled > > > > > > > > >> > > > >>>> > > > > > > > > >> and > > > > > > > > >> > > > >>>> > > > > > > > > >> > > it > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I > > > wasn't > > > > > > > aware > > > > > > > > >> > > > segment.ms=0 > > > > > > > > >> > > > >>>> == > > > > > > > > >> > > > >>>> > > > > > disabled, > > > > > > > > >> > > > >>>> > > > > > > > > but I > > > > > > > > >> > > > >>>> > > > > > > > > >> > > think > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that is also > > > unintuitive. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an > > > argument > > > > > for > > > > > > > > >> keeping > > > > > > > > >> > > > these > > > > > > > > >> > > > >>>> two > > > > > > > > >> > > > >>>> > > > > > > > configurations > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they > > > are so > > > > > > > > >> similar. I > > > > > > > > >> > > > agree > > > > > > > > >> > > > >>>> that > > > > > > > > >> > > > >>>> > 0 > > > > > > > > >> > > > >>>> > > > was an > > > > > > > > >> > > > >>>> > > > > > > > > >> > unfortunate > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > choice., > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > best, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > Colin > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 > at > > > > > 11:38 AM > > > > > > > > >> Colin > > > > > > > > >> > > > McCabe > > > > > > > > >> > > > >>>> < > > > > > > > > >> > > > >>>> > > > > > > > > cmcc...@apache.org> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, > 2018, > > > at > > > > > 17:47, > > > > > > > > >> > xiongqi > > > > > > > > >> > > wu > > > > > > > > >> > > > >>>> wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for > > > comments. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline > > reply > > > > > below. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) > > Wu > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, > > 2018 > > > at > > > > > 5:24 > > > > > > > PM > > > > > > > > >> > Colin > > > > > > > > >> > > > >>>> McCabe < > > > > > > > > >> > > > >>>> > > > > > > > > >> cmcc...@apache.org> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for > this > > > KIP. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name > seems a > > > bit > > > > > > > > >> ambiguous. > > > > > > > > >> > Our > > > > > > > > >> > > > >>>> > compaction > > > > > > > > >> > > > >>>> > > > > > > policies > > > > > > > > >> > > > >>>> > > > > > > > > are > > > > > > > > >> > > > >>>> > > > > > > > > >> > > already > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, > > after > > > > > all. It > > > > > > > > >> seems > > > > > > > > >> > > like > > > > > > > > >> > > > >>>> this > > > > > > > > >> > > > >>>> > > > change is > > > > > > > > >> > > > >>>> > > > > > > > > focused > > > > > > > > >> > > > >>>> > > > > > > > > >> > > around > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > adding > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “ > > > > > max.compaction.lag.ms > > > > > > > ." > > > > > > > > >> > Perhaps > > > > > > > > >> > > > >>>> the KIP > > > > > > > > >> > > > >>>> > > > title > > > > > > > > >> > > > >>>> > > > > > > should > > > > > > > > >> > > > >>>> > > > > > > > > be > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > something > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add > > maximum > > > > > > > compaction > > > > > > > > >> lag > > > > > > > > >> > > > time"? > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> > > sure. > > > I > > > > > will > > > > > > > > >> change > > > > > > > > >> > the > > > > > > > > >> > > > >>>> title. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active > > > segment is > > > > > > > forced > > > > > > > > >> to > > > > > > > > >> > > roll > > > > > > > > >> > > > >>>> when > > > > > > > > >> > > > >>>> > > > either " > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms > > > " > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or " > > segment.ms" > > > ( > > > > > > > > >> log.roll.ms > > > > > > > > >> > and > > > > > > > > >> > > > >>>> > > > log.roll.hours) > > > > > > > > >> > > > >>>> > > > > > > has > > > > > > > > >> > > > >>>> > > > > > > > > >> > reached. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the > > > > > > > > max.compaction.lag.ms > > > > > > > > >> is > > > > > > > > >> > > low, > > > > > > > > >> > > > >>>> it > > > > > > > > >> > > > >>>> > seems > > > > > > > > >> > > > >>>> > > > like > > > > > > > > >> > > > >>>> > > > > > > > > segments > > > > > > > > >> > > > >>>> > > > > > > > > >> > will > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > be > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very > > > > > frequently. > > > > > > > This > > > > > > > > >> can > > > > > > > > >> > > be a > > > > > > > > >> > > > >>>> > source of > > > > > > > > >> > > > >>>> > > > > > > problems > > > > > > > > >> > > > >>>> > > > > > > > > in > > > > > > > > >> > > > >>>> > > > > > > > > >> the > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating > > > many > > > > > > > > different > > > > > > > > >> > small > > > > > > > > >> > > > log > > > > > > > > >> > > > >>>> > segments > > > > > > > > >> > > > >>>> > > > > > > > consumes > > > > > > > > >> > > > >>>> > > > > > > > > a > > > > > > > > >> > > > >>>> > > > > > > > > >> > huge > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster > > resources. > > > > > > > > Therefore, > > > > > > > > >> I > > > > > > > > >> > > would > > > > > > > > >> > > > >>>> > suggest > > > > > > > > >> > > > >>>> > > > > > adding a > > > > > > > > >> > > > >>>> > > > > > > > > >> > > broker-level > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration > > > which > > > > > allows > > > > > > > > us > > > > > > > > >> to > > > > > > > > >> > > set > > > > > > > > >> > > > a > > > > > > > > >> > > > >>>> > minimum > > > > > > > > >> > > > >>>> > > > value > > > > > > > > >> > > > >>>> > > > > > > for > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > max.compaction.lag.ms. > > > > > If > > > > > > > > we > > > > > > > > >> let > > > > > > > > >> > > > >>>> users set > > > > > > > > >> > > > >>>> > it > > > > > > > > >> > > > >>>> > > > on a > > > > > > > > >> > > > >>>> > > > > > > > > >> per-topic > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > basis, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could > > set > > > a > > > > > value > > > > > > > of > > > > > > > > >> 1 ms > > > > > > > > >> > > or > > > > > > > > >> > > > >>>> > something, > > > > > > > > >> > > > >>>> > > > and > > > > > > > > >> > > > >>>> > > > > > > > cause > > > > > > > > >> > > > >>>> > > > > > > > > >> > chaos. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> > this > > > > > applies to > > > > > > > > >> > > > segment.ms > > > > > > > > >> > > > >>>> as > > > > > > > > >> > > > >>>> > well. > > > > > > > > >> > > > >>>> > > > > > Today > > > > > > > > >> > > > >>>> > > > > > > > > users > > > > > > > > >> > > > >>>> > > > > > > > > >> can > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > set " > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to > a > > > very > > > > > low > > > > > > > > >> value, > > > > > > > > >> > and > > > > > > > > >> > > > >>>> cause a > > > > > > > > >> > > > >>>> > > > frequent > > > > > > > > >> > > > >>>> > > > > > > > > >> rolling of > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > active > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this > is > > > an > > > > > > > existing > > > > > > > > >> > > problem > > > > > > > > >> > > > >>>> with > > > > > > > > >> > > > >>>> > > > > > segment.ms. > > > > > > > > >> > > > >>>> > > > > > > > > >> However, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > that > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that > we > > > > > shouldn't > > > > > > > > fix > > > > > > > > >> > it. > > > > > > > > >> > > As > > > > > > > > >> > > > >>>> you > > > > > > > > >> > > > >>>> > noted, > > > > > > > > >> > > > >>>> > > > > > there > > > > > > > > >> > > > >>>> > > > > > > > > will > > > > > > > > >> > > > >>>> > > > > > > > > >> be > > > > > > > > >> > > > >>>> > > > > > > > > >> > > more > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these > > > > > topic-level > > > > > > > > >> > retention > > > > > > > > >> > > > >>>> > settings as > > > > > > > > >> > > > >>>> > > > a > > > > > > > > >> > > > >>>> > > > > > > result > > > > > > > > >> > > > >>>> > > > > > > > > of > > > > > > > > >> > > > >>>> > > > > > > > > >> > GDPR. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > It > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that > > > > > pre-existing > > > > > > > > >> > problems > > > > > > > > >> > > > >>>> will > > > > > > > > >> > > > >>>> > cause > > > > > > > > >> > > > >>>> > > > more > > > > > > > > >> > > > >>>> > > > > > > > > trouble. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems > > > relatively > > > > > > > > >> > > straightforward > > > > > > > > >> > > > >>>> here -- > > > > > > > > >> > > > >>>> > > > add a > > > > > > > > >> > > > >>>> > > > > > > > > >> broker-level > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum > segment.ms > > > that > > > > > > > > overrides > > > > > > > > >> > > > >>>> per-topic > > > > > > > > >> > > > >>>> > > > minimums. > > > > > > > > >> > > > >>>> > > > > > We > > > > > > > > >> > > > >>>> > > > > > > > can > > > > > > > > >> > > > >>>> > > > > > > > > >> also > > > > > > > > >> > > > >>>> > > > > > > > > >> > > fail > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful > error > > > > > message > > > > > > > > when > > > > > > > > >> > > someone > > > > > > > > >> > > > >>>> > attempts > > > > > > > > >> > > > >>>> > > > to > > > > > > > > >> > > > >>>> > > > > > set > > > > > > > > >> > > > >>>> > > > > > > an > > > > > > > > >> > > > >>>> > > > > > > > > >> > invalid > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, > the > > > > > minimum > > > > > > > of " > > > > > > > > >> > > > >>>> > > > max.compaction.lag.ms" > > > > > > > > >> > > > >>>> > > > > > > > should > > > > > > > > >> > > > >>>> > > > > > > > > >> be > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the > > > minimum of > > > > > " > > > > > > > > >> > segment.ms > > > > > > > > >> > > ". > > > > > > > > >> > > > >>>> Since > > > > > > > > >> > > > >>>> > > > today the > > > > > > > > >> > > > >>>> > > > > > > > > >> minimum of > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, " > > > > > > > max.compaction.lag.ms" > > > > > > > > >> also > > > > > > > > >> > > > >>>> starts > > > > > > > > >> > > > >>>> > with > > > > > > > > >> > > > >>>> > > > 1. "0" > > > > > > > > >> > > > >>>> > > > > > > > means > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > disable. I > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as > > > disable, > > > > > but > > > > > > > it > > > > > > > > is > > > > > > > > >> > hard > > > > > > > > >> > > > to > > > > > > > > >> > > > >>>> > define > > > > > > > > >> > > > >>>> > > > the > > > > > > > > >> > > > >>>> > > > > > > > meaning > > > > > > > > >> > > > >>>> > > > > > > > > >> of 0 > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > because > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > we > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll > > the > > > > > active > > > > > > > > >> segment > > > > > > > > >> > > > >>>> > immediately. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair > point. > > > We > > > > > should > > > > > > > > >> make > > > > > > > > >> > 0 = > > > > > > > > >> > > > >>>> > disable, to > > > > > > > > >> > > > >>>> > > > be > > > > > > > > >> > > > >>>> > > > > > > > > >> consistent > > > > > > > > >> > > > >>>> > > > > > > > > >> > > with > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > the > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that > an > > > > > > > alternative > > > > > > > > >> > > > >>>> configuration > > > > > > > > >> > > > >>>> > is to > > > > > > > > >> > > > >>>> > > > use > > > > > > > > >> > > > >>>> > > > > > -1 > > > > > > > > >> > > > >>>> > > > > > > > as > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > "disabled" > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > and > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0 > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as > "immediate > > > > > > > compaction". > > > > > > > > >> > > Because > > > > > > > > >> > > > >>>> > compaction > > > > > > > > >> > > > >>>> > > > lag > > > > > > > > >> > > > >>>> > > > > > is > > > > > > > > >> > > > >>>> > > > > > > > > still > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > determined > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on > > > > > > > > min.compaction.lag > > > > > > > > >> and > > > > > > > > >> > > how > > > > > > > > >> > > > >>>> long > > > > > > > > >> > > > >>>> > to > > > > > > > > >> > > > >>>> > > > roll > > > > > > > > >> > > > >>>> > > > > > an > > > > > > > > >> > > > >>>> > > > > > > > > active > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > segment, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag > for > > > > > > > compaction > > > > > > > > is > > > > > > > > >> > > > >>>> undetermined > > > > > > > > >> > > > >>>> > if > > > > > > > > >> > > > >>>> > > > we > > > > > > > > >> > > > >>>> > > > > > use > > > > > > > > >> > > > >>>> > > > > > > > "0". > > > > > > > > >> > > > >>>> > > > > > > > > >> On > > > > > > > > >> > > > >>>> > > > > > > > > >> > the > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > other > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can > > > > > already set > > > > > > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio" > > > > > > > > >> > > > >>>> > > > > > > > to > > > > > > > > >> > > > >>>> > > > > > > > > >> > achieve > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > the > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So > here > > we > > > > > choose > > > > > > > > "0" > > > > > > > > >> as > > > > > > > > >> > > > >>>> "disabled". > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer > > -1 > > > to > > > > > be > > > > > > > the > > > > > > > > >> > invalid > > > > > > > > >> > > > >>>> setting. > > > > > > > > >> > > > >>>> > > > > > Treating > > > > > > > > >> > > > >>>> > > > > > > 0 > > > > > > > > >> > > > >>>> > > > > > > > > >> > > differently > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > than > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems > strange > > > to me. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my > > > previous > > > > > > > > >> comment, I > > > > > > > > >> > > am > > > > > > > > >> > > > >>>> not > > > > > > > > >> > > > >>>> > > > strongly > > > > > > > > >> > > > >>>> > > > > > > > against, > > > > > > > > >> > > > >>>> > > > > > > > > >> but > > > > > > > > >> > > > >>>> > > > > > > > > >> > 0 > > > > > > > > >> > > > >>>> > > > > > > > > >> > > is > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > not a > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid > > configuration > > > in > > > > > my > > > > > > > > >> option. > > > > > > > > >> > So > > > > > > > > >> > > I > > > > > > > > >> > > > >>>> use > > > > > > > > >> > > > >>>> > "0" as > > > > > > > > >> > > > >>>> > > > > > > disabled > > > > > > > > >> > > > >>>> > > > > > > > > >> state. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best, > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, > > > 2018, > > > > > at > > > > > > > > 15:04, > > > > > > > > >> > > > xiongqi > > > > > > > > >> > > > >>>> wu > > > > > > > > >> > > > >>>> > wrote: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE > for > > > this > > > > > KIP. > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > < > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > < > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > < > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > >> > > > >>>> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > > > > >> > > > >>>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > < > > > > > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > > > >>>> > < > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354 > > > > > > > > >> > >>>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > %3A+Time-based+log+compaction+policy > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > Implementation: > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > < > > > > > > > > >> https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > > >>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > < > > > > > > > > >> > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > < > > > > > > > > >> https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > >>> > > > > > > > > >> > > > >>>> > > > > > > > > >> < > > > > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>> > > > > > > > > >> > > > >>>> > > > > > > > < > > > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611> > > > > > > > > >> > > > >>>> > > > > > < > > > https://github.com/apache/kafka/pull/5611 > > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611 > > >>>>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi > > > (Wesley) Wu > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > -- > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, > > Melbourne > > > VIC > > > > > 3000 > > > > > > > > >> > > Australia > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 > 017 > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > -- > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Brett Rann > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne > > VIC > > > > > 3000 > > > > > > > > >> Australia > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017 > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > -- > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > Brett Rann > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne > VIC > > > 3000 > > > > > > > > >> Australia > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017 > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > > > >> > > > >>>> > > > > > > > > > > > >> > > > >>>> > > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
