Doyeon Kim created KAFKA-21192:
----------------------------------
Summary: Skip deleting remote segments when the leader epoch cache
does not cover the remote log range
Key: KAFKA-21192
URL: https://issues.apache.org/jira/browse/KAFKA-21192
Project: Kafka
Issue Type: Bug
Reporter: Doyeon Kim
Assignee: Doyeon Kim
When the leader epoch cache of a leader does not cover its remote log range
[logStartOffset, localLogStartOffset), which can happen to replicas hit by
KAFKA-17249, the cleanup of unreferenced epochs in RemoteLogManager treats the
remote segments of the missing epochs as unreferenced and deletes them. The
remote tier is shared, so the data is lost for every replica.
RemoteLogManager should skip this deletion while the cache is incomplete. PR
#23569 adds RemoteLogManager#isLeaderEpochCacheMissingRemoteEntries, which can
be reused for the check.
Open questions:
- Such a leader also uploads its incomplete checkpoint with every new segment,
so followers that later build their aux state from those segments inherit the
gap. Should it skip copying segments too? The cost is that local segments
cannot be deleted until they are copied.
- Skipping the deletion does not repair the leader, which still cannot serve
the remote range. The missing entries could be filled from the checkpoint of
the latest remote segment, if a healthy leader uploaded it, or rebuilt from the
segment leader epochs in the remote log metadata. Both are larger changes.
- Filtering such replicas out of leader election would need the follower to
report the state of its cache, which is a protocol change. With KAFKA-21191 , a
broken follower cannot rejoin the ISR before it is rebuilt anyway, and when the
broken replica is the only one in the ISR, filtering it out would make the
partition unavailable.
Discussion: https://github.com/apache/kafka/pull/23569#discussion_r4100224664
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
