kamalcph commented on code in PR #21089:
URL: https://github.com/apache/kafka/pull/21089#discussion_r2623538161
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -261,6 +268,12 @@ public static ConfigDef configDef() {
atLeast(1),
LOW,
REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES_DOC)
+ .define(REMOTE_LOG_INDEX_FILE_CACHE_TTL_MS_PROP,
Review Comment:
can the config be marked as internal until the KIP gets approved?
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -92,6 +92,13 @@ public final class RemoteLogManagerConfig {
"from remote storage in the local storage.";
public static final long
DEFAULT_REMOTE_LOG_INDEX_FILE_CACHE_TOTAL_SIZE_BYTES = 1024 * 1024 * 1024L;
+ public static final String REMOTE_LOG_INDEX_FILE_CACHE_TTL_MS_PROP =
"remote.log.index.file.cache.ttl.ms";
+ public static final String REMOTE_LOG_INDEX_FILE_CACHE_TTL_MS_DOC = "The
maximum time in milliseconds an index file entry can remain in the cache " +
+ "after its last access. After this duration, the entry will be
evicted even if there is available space. " +
+ "This helps prevent old index files from sitting in cache
indefinitely, especially useful for backfill workloads. " +
+ "Set to -1 to disable time-based eviction (default behavior uses
only size-based eviction).";
+ public static final long DEFAULT_REMOTE_LOG_INDEX_FILE_CACHE_TTL_MS = -1L;
Review Comment:
This is a bug. The `remote-log-index-cache` holds the expired indexes. Only
the leader of a partition deletes the remote-log-segment and clears any cached
associated index files.
The stale index files can still exist on the previous leader or when read
from replica feature gets enabled.
We can mark this config as internal and configure the default TTL to few hrs
(1 hr) to enable this feature. If the index files gets removed / evicted, then
it gets re-fetched again from remote storage when accessed.
1 hr is sufficient amount of time for the client to read a partition /
segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
