Gaurav Narula created KAFKA-19200:
-------------------------------------
Summary: Indexes should be sanity checked on startup
Key: KAFKA-19200
URL: https://issues.apache.org/jira/browse/KAFKA-19200
Project: Kafka
Issue Type: Bug
Affects Versions: 4.0.0, 3.9.0
Reporter: Gaurav Narula
KAFKA-7283 removed sanity checks for indexes on broker startup as we
thought it doesn't add much benefit. It turns out that log segment
corruption may be independent of an index corruption.
An index corruption when not caught early is quite tricky to debug. We observed
the following in production:
A corruption lead to a timeindex file on disk to be mostly filled with {{0}}s
at the end. This file is then loaded in memory such that {{DirectByteBuffer}}'s
{{position=limit=10485756}}. Note that this is 4 bytes short of 10MiB, the
configured max index size.
At this point, the log segment is eligible to be rolled as {{TimeIndex#isFull}}
will return {{true}}. We observe that the roll is attempted from 2 code paths:
1. ReplicaFetcherThread attempts to roll the log segment as it tries to append
records
2. LogCleanerThread attempts to roll the log segment as it tries to clean
segments that breach retentionMs
In both scenarios, {{LogSegment#onBecomeInactiveSegment}} is invoked which in
turn invokes {{TimeIndex#maybeAppend(long timestamp, long offset, boolean
skipFullCheck)}} with {{skipFullCheck=true}}, causing an append to an already
full TimeIndex which fails by throwing a {{BufferOverException}}.
For (1), the exception causes the partition to be marked as failed, thereby
causing an under-replicated partition. For (2), the LogCleanerThread shuts
down, potentially causing a leak as other segments which are eligible for
cleanup aren't cleaned up.
We should therefore reintroduce sanity checks on startup for indexes in
{{LogSegment#sanityCheck}}, as that is invoked regardless of an unclean
shutdown and it attempts to re-create the indexes if corruption is diagnosed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
