Hi Colin, Yes, the impact of "losing" entries in the LeaderEpoch file is more round-trips for OffsetForLeaderEpoch. There is JIRA for the log cleaner: https://issues.apache.org/jira/browse/KAFKA-6780, and to investigate if there is actual possibility of losing committed records due to cleaning further than high watermark. In any case, this KIP does not make it any more or less likely.
Thanks, Anna On Tue, Apr 17, 2018 at 11:53 AM, Colin McCabe <cmcc...@apache.org> wrote: > Thanks, Anna, this looks great. > > From the KIP: > > > Impact of topic compaction > > > > The proposed solution requires that we preserve history in > > LeaderEpochSequence file. Note that this is also required in the > current > > implementation if we want to guarantee no log divergence. The only > reason > > for "losing" entries in LeaderEpoch file is if we actually lose > > LeaderEpoch file and have to rebuild it from the log. If we delete all > > offsets for a particular epoch for some topic partition, we may miss > some > > entries in the LeaderEpochSequence file. > > > > We will not do any changes to compaction logic in this KIP, but here is > > possible fixes to compaction logic: > > > > Leave a tombstone in the log if we delete all offsets for some > epoch, > > so that LeaderEpoch file can be rebuilt > > Do not compact further than persistent HW. > > Sorry if this has been answered before (I didn't find it in the DISCUSS > thread) but what is the impact of "losing" these entries in the LeaderEpoch > file? I suppose it would mean that more round-trips for > OffsetForLeaderEpoch might be required during a leader change. That's the > only impact, right? > > best, > Colin > > On Mon, Apr 16, 2018, at 09:48, Ismael Juma wrote: > > Thanks for the detailed KIP. +1 (binding) > > > > Ismael > > > > On Sat, Apr 14, 2018 at 3:54 PM, Anna Povzner <a...@confluent.io> wrote: > > > > > Hi All, > > > > > > > > > I would like to start the vote on KIP-279: Fix log divergence between > > > leader and follower after fast leader fail over. > > > > > > > > > For reference, here's the KIP wiki: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 279%3A+Fix+log+divergence+between+leader+and+follower+ > > > after+fast+leader+fail+over > > > > > > > > > > > > and discussion thread: > > > https://www.mail-archive.com/dev@kafka.apache.org/msg86753.html > > > > > > > > > Thanks, > > > > > > Anna > > > >
