Hi Murali,

Overall LGTM. It would be my honor to give +1 on the vote thread :)

One nit: what happens if the follower's segments don't match the leader's?
For example, offsets 0-100 are in two segments on the leader but in a
single segment on the follower. Which upload time should the follower pick?

Best,
Chia-Ping

On Wed, Nov 18, 2026 03:14 AM, Muralidhar Basani via dev <
[email protected]> wrote:

> Hi all,
>
> I have updated the KIP based on the feedback so far:
>
> - Compatibility: added a "Disk sizing" section. For backfill data, local
> disk grows by about local.retention.ms * write rate per partition replica,
> with local.retention.bytes as the way to cap it.
> - Proposed Changes: clarified where the upload time comes from, and that
> local time-based deletion pauses while the remote log metadata cache is
> initializing, with no fallback to record timestamps.
> - Rejected Alternatives: added min(record timestamp, upload time).
>
> Please take a look and let me know if there are further comments.
>
> Thanks,
> Murali
>
> On Fri, Nov 13, 2026 at 10:29 PM Muralidhar Basani <
> [email protected]> wrote:
>
> > Hi Viktor,
> >
> > Thanks for the review, and nice point on disk sizing.
> >
> > Unless there are objections, I shall keep the uploadTime as base, add the
> > sizing note to compatibility, and move min(..) to rejected alternatives.
> >
> > I will update KIP.
> >
> > Chia, pls let me know if this addresses the concern you raised on Case C.
> >
> > Thanks,
> > Murali
> >
> > On Thu, Nov 12, 2026 at 11:32 AM Viktor Somogyi-Vass <
> > [email protected]> wrote:
> >
> >> Hi Murali,
> >>
> >> I also agree with Chia-Ping, taking uploadTime as the new base for
> >> local.retention.ms would generally increase local disk usage which
> might
> >> be surprising and might have disk sizing consequences as well. So the
> >> extra local disk required would be local.retention.ms * upload rate, I
> >> think we can add this to the KIP. I don't think that changing behavior
> is a
> >> blocker if it results in a more predictable behavior.
> >> I was also considering min(maxTimestamp, uploadTime), but I think that
> >> overall it would complicate the config itself, which I would not take
> >> compared to uploadTime only as that would create a common, broker-bound
> and
> >> predictable interpretation for local.retention.ms.
> >>
> >> Best,
> >> Viktor
> >>
> >> On Thu, Nov 5, 2026 at 10:03 PM Muralidhar Basani via dev <
> >> [email protected]> wrote:
> >>
> >>> Thanks Chia for the review.
> >>>
> >>> 1. Case C: Yes, this is a behavior change. It only affects topics that
> >>> set
> >>> local.retention.ms lower than the age of the backfilled data.
> >>> With the default (-2 for local.retention.ms, means use the same value
> as
> >>> retention.ms), backfilled data isn't deleted right after upload today
> >>> either.
> >>> For example, with retention.ms = 7 days and local.retention.ms left at
> >>> its
> >>> default of -2, local retention is also 7 days. Data stays on local disk
> >>> as
> >>> long as it's kept at all.
> >>>
> >>> I see 2 options.
> >>>
> >>> a. Keep upload-time semantics and document the change in the upgrade
> >>> notes.
> >>> Operators who want to bound local disk can set local.retention.bytes:
> it
> >>> is
> >>> per partition, ignores timestamps, and deletes the oldest uploaded
> >>> segments
> >>> once the limit is exceeded. For a one-off backfill, they can also lower
> >>> local.retention.ms on the topic temporarily.
> >>>
> >>> Ex for bytes : As the limit applies to each partition separately, with
> >>> 5gb
> >>> per partition, and with 10 partitions, broker can hold upto 50gb
> locally.
> >>> As it only looks at size, it will ignore the timestamps then.
> >>>
> >>> b. What do you think of an alternative which is to measure from
> >>> min(record
> >>> timestamp, upload time). That keeps today's backfill behavior and still
> >>> fixes future timestamps, but local.retention.ms would no longer mean
> the
> >>> same thing for all data.
> >>>
> >>> Which one do you think fits operators better?
> >>>
> >>> 2. Metadata cache initializing: No, we won't fall back. RLM tasks skip
> >>> their run until RemoteLogMetadataManager#isReady is true, so
> >>> highestOffsetInRemoteStorage stays -1 and the upload-before-delete
> guard
> >>> blocks time-based local deletion.
> >>> Basically deletion is paused until cache is ready. So the fallback
> isn't
> >>> triggered.
> >>>
> >>> Pls let me know your thoughts.
> >>>
> >>> Thanks,
> >>> Murali
> >>>
> >>> On Thu, Nov 5, 2026 at 7:06 PM Chia-Ping Tsai via dev <
> >>> [email protected]>
> >>> wrote:
> >>>
> >>> > Hi Murali,
> >>> >
> >>> > I'm worried about Case C: "the local copy is deleted around 11:05, 1
> >>> hour
> >>> > after the upload."
> >>> >
> >>> > Would this surprise operators who assume backfilled data is removed
> >>> right
> >>> > after upload? I guess the suitable approach for them is to use
> >>> > local.retention.bytes?
> >>> >
> >>> > Also, what happens if the RemoteLogMetadataCache is still
> initializing
> >>> > when the broker has just started? Will we always take the fallback?
> >>> >
> >>> > Best,
> >>> > Chia-Ping
> >>> >
> >>> >
> >>> >
> >>> > On 2026-10-23 18:30 UTC Muralidhar Basani via dev wrote:
> >>> > > Hi all,
> >>> > >
> >>> > > I would like to start a discussion about tiered storage topics and
> >>> how
> >>> > > local segments are deleted by time, once they are uploaded to
> remote
> >>> > > storage.
> >>> > >
> >>> > >
> >>> >
> >>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/45197
> 9130/KIP-1385+Measure+local.retention.ms+from+remote+
> upload+time+instead+of+record+timestamps
> >>> > >
> >>> > > Looking forward to your thoughts on this approach.
> >>> > >
> >>> > > Thanks,
> >>> > > Murali
> >>> > >
> >>> >
> >>>
> >>
>

Reply via email to