nandini12396 commented on PR #21088:
URL: https://github.com/apache/kafka/pull/21088#issuecomment-3662330907
Thanks for the question and review @kamalcph!
Yes, reader threads can hold index entries for longer than 10 seconds in
production. This happens when remote storage fetches are slow, during GC
pauses, or under high load. The 10-second delay only delays file deletion, but
doesn't prevent threads from holding stale references to marked entries.
I was able to reproduce this consistently in test env by reducing the size
of `remote_index_cache_size_bytes` to force frequent evictions and triggering
TS fetches.
Here is one of the recent stack traces showing the exception:
```
[2025-12-15 08:52:12,042] ERROR Error occurred while reading the remote data
for test-stream-nandini_normal_5cc7-2 (kafka.log.remote.RemoteLogReader)
java.lang.IllegalStateException: This entry is marked for cleanup
at
org.apache.kafka.storage.internals.log.RemoteIndexCache$Entry.lookupOffset(RemoteIndexCache.java:559)
at
org.apache.kafka.storage.internals.log.RemoteIndexCache.lookupOffset(RemoteIndexCache.java:448)
at
kafka.log.remote.RemoteLogManager.lookupPositionForOffset(RemoteLogManager.java:1779)
at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1717)
at
kafka.log.remote.RemoteLogReader.lambda$call$0(RemoteLogReader.java:72)
at com.yammer.metrics.core.Timer.time(Timer.java:91)
at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:72)
at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:36)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
