Hi Chia, Thanks for the review.
As only leader uploads files, a follower's file can cover several of the leader's uploaded files. The follower uses the upload time of the last one, since that is when its whole file was available remotely. In your example, the follower's 0-100 file uses the upload time of the leader's 50-100 file. If it's the other way round (one leader file, two follower files), both follower files use the same upload time. I have updated the KIP based on this. If there are no further comments, I shall start a vote thread soon. Thanks, Murali On Thu, Nov 19, 2026 at 9:44 AM Chia-Ping Tsai <[email protected]> wrote: > Hi Murali, > > Overall LGTM. It would be my honor to give +1 on the vote thread :) > > One nit: what happens if the follower's segments don't match the leader's? > For example, offsets 0-100 are in two segments on the leader but in a > single segment on the follower. Which upload time should the follower pick? > > Best, > Chia-Ping > > On Wed, Nov 18, 2026 03:14 AM, Muralidhar Basani via dev < > [email protected]> wrote: > > > Hi all, > > > > I have updated the KIP based on the feedback so far: > > > > - Compatibility: added a "Disk sizing" section. For backfill data, local > > disk grows by about local.retention.ms * write rate per partition > replica, > > with local.retention.bytes as the way to cap it. > > - Proposed Changes: clarified where the upload time comes from, and that > > local time-based deletion pauses while the remote log metadata cache is > > initializing, with no fallback to record timestamps. > > - Rejected Alternatives: added min(record timestamp, upload time). > > > > Please take a look and let me know if there are further comments. > > > > Thanks, > > Murali > > > > On Fri, Nov 13, 2026 at 10:29 PM Muralidhar Basani < > > [email protected]> wrote: > > > > > Hi Viktor, > > > > > > Thanks for the review, and nice point on disk sizing. > > > > > > Unless there are objections, I shall keep the uploadTime as base, add > the > > > sizing note to compatibility, and move min(..) to rejected > alternatives. > > > > > > I will update KIP. > > > > > > Chia, pls let me know if this addresses the concern you raised on Case > C. > > > > > > Thanks, > > > Murali > > > > > > On Thu, Nov 12, 2026 at 11:32 AM Viktor Somogyi-Vass < > > > [email protected]> wrote: > > > > > >> Hi Murali, > > >> > > >> I also agree with Chia-Ping, taking uploadTime as the new base for > > >> local.retention.ms would generally increase local disk usage which > > might > > >> be surprising and might have disk sizing consequences as well. So the > > >> extra local disk required would be local.retention.ms * upload rate, > I > > >> think we can add this to the KIP. I don't think that changing behavior > > is a > > >> blocker if it results in a more predictable behavior. > > >> I was also considering min(maxTimestamp, uploadTime), but I think that > > >> overall it would complicate the config itself, which I would not take > > >> compared to uploadTime only as that would create a common, > broker-bound > > and > > >> predictable interpretation for local.retention.ms. > > >> > > >> Best, > > >> Viktor > > >> > > >> On Thu, Nov 5, 2026 at 10:03 PM Muralidhar Basani via dev < > > >> [email protected]> wrote: > > >> > > >>> Thanks Chia for the review. > > >>> > > >>> 1. Case C: Yes, this is a behavior change. It only affects topics > that > > >>> set > > >>> local.retention.ms lower than the age of the backfilled data. > > >>> With the default (-2 for local.retention.ms, means use the same > value > > as > > >>> retention.ms), backfilled data isn't deleted right after upload > today > > >>> either. > > >>> For example, with retention.ms = 7 days and local.retention.ms left > at > > >>> its > > >>> default of -2, local retention is also 7 days. Data stays on local > disk > > >>> as > > >>> long as it's kept at all. > > >>> > > >>> I see 2 options. > > >>> > > >>> a. Keep upload-time semantics and document the change in the upgrade > > >>> notes. > > >>> Operators who want to bound local disk can set local.retention.bytes: > > it > > >>> is > > >>> per partition, ignores timestamps, and deletes the oldest uploaded > > >>> segments > > >>> once the limit is exceeded. For a one-off backfill, they can also > lower > > >>> local.retention.ms on the topic temporarily. > > >>> > > >>> Ex for bytes : As the limit applies to each partition separately, > with > > >>> 5gb > > >>> per partition, and with 10 partitions, broker can hold upto 50gb > > locally. > > >>> As it only looks at size, it will ignore the timestamps then. > > >>> > > >>> b. What do you think of an alternative which is to measure from > > >>> min(record > > >>> timestamp, upload time). That keeps today's backfill behavior and > still > > >>> fixes future timestamps, but local.retention.ms would no longer mean > > the > > >>> same thing for all data. > > >>> > > >>> Which one do you think fits operators better? > > >>> > > >>> 2. Metadata cache initializing: No, we won't fall back. RLM tasks > skip > > >>> their run until RemoteLogMetadataManager#isReady is true, so > > >>> highestOffsetInRemoteStorage stays -1 and the upload-before-delete > > guard > > >>> blocks time-based local deletion. > > >>> Basically deletion is paused until cache is ready. So the fallback > > isn't > > >>> triggered. > > >>> > > >>> Pls let me know your thoughts. > > >>> > > >>> Thanks, > > >>> Murali > > >>> > > >>> On Thu, Nov 5, 2026 at 7:06 PM Chia-Ping Tsai via dev < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>> > Hi Murali, > > >>> > > > >>> > I'm worried about Case C: "the local copy is deleted around 11:05, > 1 > > >>> hour > > >>> > after the upload." > > >>> > > > >>> > Would this surprise operators who assume backfilled data is removed > > >>> right > > >>> > after upload? I guess the suitable approach for them is to use > > >>> > local.retention.bytes? > > >>> > > > >>> > Also, what happens if the RemoteLogMetadataCache is still > > initializing > > >>> > when the broker has just started? Will we always take the fallback? > > >>> > > > >>> > Best, > > >>> > Chia-Ping > > >>> > > > >>> > > > >>> > > > >>> > On 2026-10-23 18:30 UTC Muralidhar Basani via dev wrote: > > >>> > > Hi all, > > >>> > > > > >>> > > I would like to start a discussion about tiered storage topics > and > > >>> how > > >>> > > local segments are deleted by time, once they are uploaded to > > remote > > >>> > > storage. > > >>> > > > > >>> > > > > >>> > > > >>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/45197 > > 9130/KIP-1385+Measure+local.retention.ms+from+remote+ > > upload+time+instead+of+record+timestamps > > >>> > > > > >>> > > Looking forward to your thoughts on this approach. > > >>> > > > > >>> > > Thanks, > > >>> > > Murali > > >>> > > > > >>> > > > >>> > > >> > > >
