Xiaobing Fang created KAFKA-21367:
-------------------------------------
Summary: Stale recovery checkpoint can overwrite truncation
checkpoint and skip log recovery
Key: KAFKA-21367
URL: https://issues.apache.org/jira/browse/KAFKA-21367
Project: Kafka
Issue Type: Bug
Components: core
Reporter: Xiaobing Fang
h3. Problem
{\{LogManager.checkpointRecoveryOffsetsInDir}} collects recovery points before
calling \{{OffsetCheckpointFile.write}}. The lock in \{{CheckpointFile.write}}
serializes file writes, but does not cover recovery point sampling.
A periodic checkpoint can therefore overwrite a checkpoint written after log
truncation:
# A checkpoint task samples \{{recoveryPoint = 10}}, then pauses before writing.
# Another thread truncates the log to offset 0, lowering its recovery point to
0, and writes a checkpoint containing 0.
# The first task resumes and writes its previously sampled value of 10.
The persisted recovery point is now ahead of the actual flushed boundary. This
affects checkpoints following both \{{truncateTo}} and
\{{truncateFullyAndStartAt}}.
h3. Recovery impact
After truncation, new records can be appended and the log can roll before the
asynchronous flush completes. If an unclean shutdown leaves a damaged batch in
an earlier segment, the stale checkpoint can cause recovery to skip that
segment.
For example, with segment base offsets \{0, 1} and a stale checkpoint of 10,
\{{LogSegments.values(10, Long.MAX_VALUE)}} starts at \{{floorKey(10) = 1}}.
Segment 0 is not recovered. With the correct checkpoint of 0, both segments are
included.
A corrupt batch in the skipped segment can survive startup and fail subsequent
record validation with \{{CorruptRecordException}}. The race does not itself
corrupt record bytes; it can prevent recovery from detecting and truncating
damaged records.
h3. Reproduction
The checkpoint overwrite was reproduced deterministically on trunk at
\{{2bd1d20063ce7e0843bb1482f7252b67c2a40421}}. Tests for both truncation paths
read back 10 from the checkpoint file instead of the expected 0.
The recovery consequence was separately reproduced using controlled CRC
corruption, a segment roll, a pending asynchronous flush, and reopening the log
without a clean-shutdown marker. With the stale checkpoint, the damaged earlier
segment was skipped. With the correct checkpoint, recovery detected the
corruption and truncated the log.
CRC corruption was injected to test the recovery behavior; this was not a real
power-loss experiment.
h3. Proposed fix
Serialize recovery point sampling and checkpoint writing using the same
per-directory checkpoint object. This prevents an older snapshot from being
written after a newer truncation checkpoint, without changing the checkpoint
format.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
