Hi Lijun, A few comments on the upgrade path:
1. We could start producing keyed records to the __remote_log_metadata topic and optional tombstone records based on the metadata version, while leaving the existing topic configuration unchanged. For example: metadataCache.metadataVersion().isRemoteLogTombstoneRecordsSupported() 2. We can provide a verification script that checks whether all records in the metadata topic have non-null keys before changing the topic configuration. Once verified, users can update the topic's cleanup policy and other compaction-related configurations using kafka-configs.sh. We should document this process in the upgrade notes. 3. Newly created __remote_log_metadata topics can be configured with the cleanup policy set to compact. Regarding the remote log metadata key: The proposal looks good to me. We could also consider removing topicName from the key and using KRaftMetadataCache to resolve the topic name from the topicId when needed. That said, keeping topicName in the key is also reasonable if the overhead is only a few extra bytes and avoids an additional lookup into KRaftMetadataCache. Thanks, Kamal On Fri, Jul 10, 2026 at 7:24 AM Kamal Chandraprakash < [email protected]> wrote: > The approach mentioned in the KIP also handles the correctness and retries > for any failed segment uploads. > It is LGTM. > > To reduce the memory used by RemoteLogMetadataCache: > > - Shall we call removeFromEndOffsetIndex once the segment copy is finished > successfully? (in > RemotePartitionMetadataStore#handleRemoteLogSegmentMetadataUpdate method > [1]) > - TopicName can be large upto 249 chars. I'm not sure how to retrieve the > topicName from topicId. > > [1]: > https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemotePartitionMetadataStore.java?L80 > > Thanks, > Kamal > > On Fri, Jul 10, 2026 at 6:48 AM Kamal Chandraprakash < > [email protected]> wrote: > >> > Went over the code again, it is not clear to me how do we handle the >> failed segment entries for the below case(s): >> >> Got it. The failed segment entries get eventually cleared when the >> leaderEpoch itself is gone. >> >> One question that remains is: >> - Are we fine with increased memory usage by RemoteLogMetadataCache? >> - Another way is to configure the cleanup policy to [delete, compact] to >> eventually delete any stale metadata entries. >> This adds dependency that the user should configure the max upper >> bound. (eg) 45 days, assuming all the user >> topic retention is less than 30 days. >> >> >> On Fri, Jul 10, 2026 at 6:31 AM Kamal Chandraprakash < >> [email protected]> wrote: >> >>> Hi Lijun, >>> >>> Please find my comments inline: >>> >>> > the current key design also provides very >>> little context. >>> >>> The Compaction process loads the record keys in memory. >>> Keeping the key short helps in reducing the memory usage. >>> Also, maintaining the (endOffset -> (brokerLeaderEpoch -> >>> RemoteLogMetadataKey)) in >>> RemoteLogMetadataCache might take up more space. >>> >>> I'm fine with either keys: >>> >>> topicId + partition + segmentId + state >>> (or) >>> topicId + topic + partition + endOffset + brokerLeaderEpoch + optional >>> UPDATE >>> >>> > It doesn't tell users which broker (or leader epoch) >>> Id of the broker that is uploading the segment is captured in the >>> RemoteLogMetadata [1] >>> >>> > If a broker goes down unexpectedly during an upload >>> attempt, how would we later identify that failed attempt and publish a >>> tombstone for the corresponding segmentId? >>> >>> In your approach, after receiving the tombstone event, we are clearing >>> up any stale metadata events. >>> Went over the code again, it is not clear to me how do we handle the >>> failed segment entries for the below case(s): >>> >>> Broker1 uploads the segment with start-offset-endOffset as 5-900 and >>> fails to upload due to leader transition. >>> Broker2 uploads the segment with start-offset-endOffset as 3-925 and >>> succeeds to upload the segment. >>> >>> How do we clean the failed segment (5-900) uploaded by Broker1? Also, >>> there is another case >>> where the Broker2 uploads the segment with an offset range of 3-850. As >>> log rotation is independent of brokers. >>> >>> [1]: >>> https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogMetadata.java?L33 >>> >>> On Fri, Jul 10, 2026 at 3:02 AM Lijun Tong <[email protected]> >>> wrote: >>> >>>> Hey Kamal, >>>> >>>> Thanks for the quick response. >>>> >>>> One concern is that upload failures don't happen exclusively while the >>>> broker is online. If a broker goes down unexpectedly during an upload >>>> attempt, how would we later identify that failed attempt and publish a >>>> tombstone for the corresponding segmentId? >>>> >>>> From an operational perspective, the current key design also provides >>>> very >>>> little context. It doesn't tell users which broker (or leader epoch) >>>> attempted to upload which log segment, making it difficult to reason >>>> about >>>> the behavior of tiered storage. In practice, it turns the remote >>>> metadata >>>> into something of a black box. >>>> >>>> I believe the original key design, >>>> TopicIdPartition:TopicName:Partition:EndOffset:BrokerLeaderEpoch, is >>>> much >>>> more self-descriptive. It clearly identifies the log segment being >>>> tracked >>>> and provides enough context for users to understand and debug the >>>> behavior >>>> of the remote metadata manager. >>>> >>>> Best, >>>> >>>> Lijun Tong >>>> >>>> Kamal Chandraprakash <[email protected]> 于2026年7月9日周四 >>>> 10:00写道： >>>> >>>> > Hi Lijun, >>>> > >>>> > Thanks for the follow up! >>>> > >>>> > > Since a new segmentId is >>>> > generated every time a broker retries uploading the same log segment, >>>> it >>>> > doesn't provide a persistent identity for that log segment. >>>> > >>>> > RemoteLogManager tries to delete the remote log segment when it fails >>>> to >>>> > upload [1]. >>>> > Only when encountering RetriableRemoteStorageException, it does not >>>> invoke >>>> > the `deleteRemoteLogSegment` call [2]. >>>> > If we extend the behaviour to issue delete remote log segments >>>> > (delete_seg_started and del_seg_finished events) in case of any upload >>>> > failures. >>>> > Does it simplify the overall approach? >>>> > >>>> > [1]: >>>> > >>>> > >>>> https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java?L1117 >>>> > [2]: >>>> > >>>> > >>>> https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java?L1113 >>>> > >>>> > >>>> > Thanks, >>>> > Kamal >>>> > >>>> > On Thu, Jul 9, 2026 at 9:32 PM Lijun Tong <[email protected]> >>>> wrote: >>>> > >>>> > > Hey Kamal, >>>> > > >>>> > > >>>> > > I reviewed your implementation of keyed events. I noticed that the >>>> key is >>>> > > designed as topicId + partition + segmentId + stateId. >>>> > > >>>> > > I don't think we've fully discussed the tradeoffs or the potential >>>> > > concurrency implications of this key design yet. >>>> > > >>>> > > My main concern is that segmentId doesn't seem stable enough to >>>> represent >>>> > > the lifecycle of a log segment in remote storage. Since a new >>>> segmentId >>>> > is >>>> > > generated every time a broker retries uploading the same log >>>> segment, it >>>> > > doesn't provide a persistent identity for that log segment. >>>> > > >>>> > > Additionally, there isn't an explicit way to determine which log >>>> segment >>>> > a >>>> > > given segmentId corresponds to. That raises the question of how we >>>> would >>>> > > publish tombstone messages for upload attempts that failed and were >>>> later >>>> > > retried. >>>> > > >>>> > > With this key design, it seems like concurrency control and cleanup >>>> logic >>>> > > could become fairly complicated. In contrast, with the current key >>>> > design, >>>> > > concurrency control is naturally straightforward, and the >>>> correctness of >>>> > > the cleanup logic is much easier to reason about and verify. >>>> > > >>>> > > >>>> > > What do you think? >>>> > > >>>> > > >>>> > > Best, >>>> > > >>>> > > Lijun Tong >>>> > > >>>> > > Kamal Chandraprakash <[email protected]> 于2026年7月7日周二 >>>> > > 02:42写道： >>>> > > >>>> > > > Hi Lijun, >>>> > > > >>>> > > > I made a first pass on the >>>> https://github.com/apache/kafka/pull/22459 >>>> > > PR. >>>> > > > Most of the changes in the schema layer may not be required. And, >>>> we >>>> > > should >>>> > > > be able to achieve the outcome with minimal changes. >>>> > > > >>>> > > > I've opened a draft proposal to consider: >>>> > > > https://github.com/apache/kafka/pull/22774. Could you take a >>>> look? >>>> > > > >>>> > > > Thanks, >>>> > > > Kamal >>>> > > > >>>> > > > On Wed, Jun 3, 2026 at 12:20 PM Lijun Tong < >>>> [email protected]> >>>> > > > wrote: >>>> > > > >>>> > > > > Hey team, >>>> > > > > >>>> > > > > I have finished the code change for this KIP, and PR is here >>>> > > > > https://github.com/apache/kafka/pull/22459. There are 2 >>>> commits in >>>> > > this >>>> > > > > PR, >>>> > > > > the first commit includes all the source code change, the second >>>> > commit >>>> > > > > includes all the test changes. >>>> > > > > I have adopted some advices from this email chain and have also >>>> > updated >>>> > > > the >>>> > > > > KIP and code accordingly. >>>> > > > > >>>> > > > > Would love to hear your thoughts. >>>> > > > > >>>> > > > > Thanks, >>>> > > > > Lijun Tong >>>> > > > > >>>> > > > > jian fu <[email protected]> 于2026年4月11日周六 18:38写道： >>>> > > > > >>>> > > > > > Hi Kamal and Lijun Tong >>>> > > > > > >>>> > > > > > Maybe another approach (Not good solution) is when we sure >>>> about >>>> > the >>>> > > > work >>>> > > > > > is complete. we send all the keys' to null to delete all the >>>> data. >>>> > > > > > It mean the keys are: >>>> > > > > > clusterid:topic:partition:offset: COPY_SEGMENT_FINISHED >>>> > > > > > clusterid:topic:partition:offset: DELETE_SEGMENT_STARTED >>>> > > > > > clusterid:topic:partition:offset: DELETE_SEGMENT_FINISHED >>>> > > > > > ---------------- >>>> > > > > > >>>> > > > > > This solution won't change the old payload in the message. >>>> Thus the >>>> > > > > > solution is somehow strange. >>>> > > > > > Or carry all complete message so that keep the latest maybe >>>> can >>>> > work. >>>> > > > > > >>>> > > > > > BTW: There are four type message need to take care: >>>> > > > > > RemoteLogSegmentMetadataRecord >>>> > > > > > RemoteLogSegmentMetadataUpdateRecord >>>> > > > > > RemotePartitionDeleteMetadataRecord >>>> > > > > > RemoteLogSegmentMetadataSnapshotRecord >>>> > > > > > >>>> > > > > > Regards >>>> > > > > > Jian >>>> > > > > > >>>> > > > > > >>>> > > > > > Lijun Tong <[email protected]> 于2026年4月7日周二 08:35写道： >>>> > > > > > >>>> > > > > > > Hi Kamal, >>>> > > > > > > >>>> > > > > > > I now see what you mean regarding the >>>> DELETE_SEGMENT_STARTED >>>> > > > handling. >>>> > > > > > > >>>> > > > > > > I was thinking that the broker's logic that currently >>>> ignores >>>> > log >>>> > > > > > segment >>>> > > > > > > messages when COPY_SEGMENT_STARTED doesn't exist might need >>>> to be >>>> > > > > updated >>>> > > > > > > for the >>>> > > > > > > new message format. Since the new message contains useful >>>> > > > > information >>>> > > > > > > like endOffset, topicID, and partition, it could provide >>>> > sufficient >>>> > > > > > context >>>> > > > > > > to help >>>> > > > > > > with the retry of deletion operations. This suggests the >>>> broker >>>> > > > might >>>> > > > > > not >>>> > > > > > > need to ignore these messages solely because of the absence >>>> of >>>> > > > > > > COPY_SEGMENT_STARTED. >>>> > > > > > > >>>> > > > > > > With the new key-based format, we would have enough >>>> metadata in >>>> > > the >>>> > > > > > > DELETE_SEGMENT_STARTED event itself to: >>>> > > > > > > - Identify the remote segment location >>>> > > > > > > - Track and retry the deletion process >>>> > > > > > > - Prevent orphaned segments in remote storage >>>> > > > > > > >>>> > > > > > > I believe this means that with the new keyed message >>>> format, we >>>> > > > could >>>> > > > > > > avoid orphaned remote segments that might occur when >>>> > > > > COPY_SEGMENT_STARTED >>>> > > > > > > is cleaned >>>> > > > > > > by the retention policy. The DELETE_SEGMENT_STARTED event >>>> would >>>> > > > carry >>>> > > > > > all >>>> > > > > > > the necessary information to complete the deletion, >>>> regardless of >>>> > > > > whether >>>> > > > > > > the >>>> > > > > > > base metadata still exists. >>>> > > > > > > >>>> > > > > > > For the old format messages, the same mechanism (ignoring >>>> when >>>> > > > > > > COPY_SEGMENT_STARTED is missing) could still apply, as those >>>> > > messages >>>> > > > > > lack >>>> > > > > > > the necessary >>>> > > > > > > information for independent processing. >>>> > > > > > > >>>> > > > > > > I'd appreciate your thoughts on this approach. >>>> > > > > > > >>>> > > > > > > Best, >>>> > > > > > > Lijun Tong >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > Kamal Chandraprakash <[email protected]> >>>> > 于2026年4月5日周日 >>>> > > > > > > 17:37写道： >>>> > > > > > > >>>> > > > > > > > Hi Lijun, >>>> > > > > > > > >>>> > > > > > > > Yes, we can get on a call to close out on this. My >>>> concern is >>>> > > that >>>> > > > if >>>> > > > > > the >>>> > > > > > > > same key is maintained for a given segment metadata >>>> records, >>>> > > > > > > > then the newer messages (COPY_FINSIHED, DELETE_STARTED) >>>> might >>>> > > > > > > > override/compact the previous COPY_STARTED events. >>>> > > > > > > > This is not about the old / new format of messages. >>>> Assume that >>>> > > all >>>> > > > > the >>>> > > > > > > > messages in the topic are in the new format and contain >>>> keys. >>>> > > > > > > > >>>> > > > > > > > From >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://docs.confluent.io/kafka/design/log_compaction.html#topic-compaction >>>> > > > > > > > : >>>> > > > > > > > >>>> > > > > > > > Topic compaction is a mechanism that allows you to retain >>>> the >>>> > > > latest >>>> > > > > > > value >>>> > > > > > > > for each message key in a topic, while discarding older >>>> values. >>>> > > It >>>> > > > > > > > guarantees that the latest value for each message key is >>>> always >>>> > > > > > retained >>>> > > > > > > > within the log of data contained in that topic, making it >>>> ideal >>>> > > for >>>> > > > > use >>>> > > > > > > > cases such as restoring state after system failure or >>>> reloading >>>> > > > > caches >>>> > > > > > > > after application restarts. >>>> > > > > > > > >>>> > > > > > > > Thanks, >>>> > > > > > > > Kamal >>>> > > > > > > > >>>> > > > > > > > On Sun, Apr 5, 2026 at 11:37 PM Lijun Tong < >>>> > > > [email protected]> >>>> > > > > > > > wrote: >>>> > > > > > > > >>>> > > > > > > > > Hi Kamal, >>>> > > > > > > > > >>>> > > > > > > > > Thanks for raising this. >>>> > > > > > > > > >>>> > > > > > > > > Currently, only the existing version of >>>> > > > > > > > > RemoteLogSegmentMetadataUpdateRecord >>>> > > > > > > > > does not include those fields. We rely on the time-based >>>> > > > retention >>>> > > > > > > policy >>>> > > > > > > > > for cleanup, and this does not impact the ability to >>>> rebuild >>>> > > the >>>> > > > > > > > > RemoteLogMetadataCache. >>>> > > > > > > > > >>>> > > > > > > > > The cache reconstruction should still work correctly >>>> because >>>> > it >>>> > > > > > depends >>>> > > > > > > > on >>>> > > > > > > > > the value, and we have not removed any fields from the >>>> value. >>>> > > > > > > > > >>>> > > > > > > > > Regarding the scenario where there are 10 remote >>>> segments and >>>> > > the >>>> > > > > > > > > __remote_log_metadata topic contains only >>>> > COPY_SEGMENT_FINISHED >>>> > > > > > events: >>>> > > > > > > > the >>>> > > > > > > > > COPY_SEGMENT_STARTED events will not be compacted in >>>> this >>>> > case, >>>> > > > > since >>>> > > > > > > > > messages with a null key are not subject to compaction. >>>> > > > > > > > > >>>> > > > > > > > > Once older-format messages are cleaned up by the >>>> time-based >>>> > > > > retention >>>> > > > > > > > > policy and compaction is enabled, records with the same >>>> key >>>> > > will >>>> > > > be >>>> > > > > > > > > compacted asynchronously and correctly. Given this, I >>>> don’t >>>> > > > believe >>>> > > > > > we >>>> > > > > > > > need >>>> > > > > > > > > to introduce a separate key for COPY_SEGMENT_STARTED >>>> events. >>>> > > > > > > > > >>>> > > > > > > > > Happy to jump on a call if it’s easier to discuss >>>> further. >>>> > > > > > > > > >>>> > > > > > > > > Best, >>>> > > > > > > > > >>>> > > > > > > > > Lijun Tong >>>> > > > > > > > > >>>> > > > > > > > > Lijun Tong <[email protected]> 于2026年4月5日周日 >>>> 10:56写道： >>>> > > > > > > > > >>>> > > > > > > > > > Hey Kamal, >>>> > > > > > > > > > >>>> > > > > > > > > > I am not very clear on what's the question you >>>> mentioned >>>> > > > above, I >>>> > > > > > am >>>> > > > > > > > > > happy to jump to a call to discuss further, and I >>>> lived in >>>> > > PST >>>> > > > > time >>>> > > > > > > > zone. >>>> > > > > > > > > > Maybe we can meet online through google meet? >>>> > > > > > > > > > >>>> > > > > > > > > > Thanks, >>>> > > > > > > > > > Lijun Tong >>>> > > > > > > > > > >>>> > > > > > > > > > Kamal Chandraprakash <[email protected]> >>>> > > > > 于2026年4月1日周三 >>>> > > > > > > > > > 02:38写道： >>>> > > > > > > > > > >>>> > > > > > > > > >> Hi Lijun, >>>> > > > > > > > > >> >>>> > > > > > > > > >> Thanks for the update! I'm still not clear on this. >>>> > > > > > > > > >> >>>> > > > > > > > > >> The RemoteLogSegmentMetadataUpdateRecord does not >>>> contain >>>> > > the >>>> > > > > > below >>>> > > > > > > > > fields >>>> > > > > > > > > >> compared to RemoteLogSegmentMetadataRecord: >>>> > > > > > > > > >> >>>> > > > > > > > > >> - startOffset >>>> > > > > > > > > >> - endOffset (will be added as a tagged field) >>>> > > > > > > > > >> - MaxTimestampMs >>>> > > > > > > > > >> - SegmentLeaderEpochs >>>> > > > > > > > > >> - SegmentSizeInBytes and >>>> > > > > > > > > >> - TxnIndexEmpty >>>> > > > > > > > > >> >>>> > > > > > > > > >> When a broker gets restarted, will it be able to >>>> rebuild >>>> > > > > > > > > >> the RemoteLogMetadataCache? Assume that there are 10 >>>> > remote >>>> > > > > > > > > >> segments and the __remote_log_metadata topic >>>> contains only >>>> > > the >>>> > > > > > > > > >> COPY_SEGMENT_FINISHED events; the >>>> COPY_SEGMENT_STARTED >>>> > event >>>> > > > > > > > > >> gets compacted as the key is the same. >>>> > > > > > > > > >> >>>> > > > > > > > > >> Do we need a separate key for the >>>> COPY_SEGMENT_STARTED >>>> > event >>>> > > > and >>>> > > > > > > > another >>>> > > > > > > > > >> key for the remaining states? >>>> > > > > > > > > >> >>>> > > > > > > > > >> Current key format: >>>> > > > TopicIdPartition:EndOffset:BrokerLeaderEpoch >>>> > > > > > > > > >> Proposed key format: >>>> > > > > > > TopicIdPartition:EndOffset:BrokerLeaderEpoch:x/y >>>> > > > > > > > > >> where >>>> > > > > > > > > >> x denotes a identifier for COPY_SEGMENT_STARTED and y >>>> > denote >>>> > > > for >>>> > > > > > all >>>> > > > > > > > the >>>> > > > > > > > > >> other events. >>>> > > > > > > > > >> >>>> > > > > > > > > >> Thanks, >>>> > > > > > > > > >> Kamal >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> On Tue, Mar 31, 2026 at 8:23 AM Lijun Tong < >>>> > > > > > [email protected] >>>> > > > > > > > >>>> > > > > > > > > >> wrote: >>>> > > > > > > > > >> >>>> > > > > > > > > >> > Hi Kamal, >>>> > > > > > > > > >> > >>>> > > > > > > > > >> > The scenario you described only happened with the >>>> old >>>> > > > version >>>> > > > > > > > > >> > RemoteLogSegmentUpdateMetadata message, since the >>>> > > endOffset >>>> > > > > will >>>> > > > > > > be >>>> > > > > > > > > >> added >>>> > > > > > > > > >> > in the new RemoteLogSegmentUpdateMetadata schema. >>>> For >>>> > the >>>> > > > > > existing >>>> > > > > > > > > >> > RemoteLogSegmentUpdateMetadata messages, we rely >>>> on the >>>> > > time >>>> > > > > > based >>>> > > > > > > > > >> > retention policy to clean up. Does that make sense? >>>> > > > > > > > > >> > >>>> > > > > > > > > >> > Best, >>>> > > > > > > > > >> > Lijun Tong >>>> > > > > > > > > >> > >>>> > > > > > > > > >> > Kamal Chandraprakash < >>>> [email protected]> >>>> > > > > > > 于2026年3月30日周一 >>>> > > > > > > > > >> > 18:14写道： >>>> > > > > > > > > >> > >>>> > > > > > > > > >> > > Hi Lijun, >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > > RemoteLogSegmentUpdateMetadata event does not >>>> have all >>>> > > the >>>> > > > > > > > > >> > > fields/attributes similar to >>>> RemoteLogSegmentMetadata >>>> > > > event. >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > > Assume that after compaction, for a segment, we >>>> have >>>> > > only >>>> > > > > > > > > >> > > COPY_SEGMENT_FINISHED records. How do you plan to >>>> > > retrieve >>>> > > > > the >>>> > > > > > > > other >>>> > > > > > > > > >> > fields >>>> > > > > > > > > >> > > after broker restart? >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > > Thanks, >>>> > > > > > > > > >> > > Kamal >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > > On Mon, Mar 30, 2026, 23:22 Lijun Tong < >>>> > > > > > [email protected] >>>> > > > > > > > >>>> > > > > > > > > >> wrote: >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > > > Hi Kamal, >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > Thanks for taking another look at the KIP. >>>> > > > > > > > > >> > > > 1. I have removed the left-over line about >>>> using >>>> > > another >>>> > > > > new >>>> > > > > > > > topic >>>> > > > > > > > > >> from >>>> > > > > > > > > >> > > the >>>> > > > > > > > > >> > > > KIP. >>>> > > > > > > > > >> > > > 2. >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > 2. Assume that the topic is enabled with >>>> > compaction >>>> > > > and >>>> > > > > > only >>>> > > > > > > > one >>>> > > > > > > > > >> > event >>>> > > > > > > > > >> > > is >>>> > > > > > > > > >> > > > > maintained per segment. If there is a >>>> transient >>>> > > error >>>> > > > in >>>> > > > > > the >>>> > > > > > > > > >> remote >>>> > > > > > > > > >> > log >>>> > > > > > > > > >> > > > > deletion, >>>> > > > > > > > > >> > > > > then the COPY_SEGMENT started / finished >>>> > events >>>> > > > > might >>>> > > > > > be >>>> > > > > > > > > >> > compacted >>>> > > > > > > > > >> > > by >>>> > > > > > > > > >> > > > > the DELETE_SEGMENT_STARTED events. If the >>>> broker >>>> > is >>>> > > > > > > restarted >>>> > > > > > > > > >> during >>>> > > > > > > > > >> > > > > this time, will there be dangling remote >>>> log >>>> > > > > segments? >>>> > > > > > > > > >> Currently, >>>> > > > > > > > > >> > > > > during restart, the broker discards the >>>> events if >>>> > it >>>> > > > > does >>>> > > > > > > not >>>> > > > > > > > > see >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > > COPY_SEGMENT_STARTED events. >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > I am glad you asked this question, I didn't >>>> mention >>>> > > this >>>> > > > > > part >>>> > > > > > > in >>>> > > > > > > > > my >>>> > > > > > > > > >> > > current >>>> > > > > > > > > >> > > > design to avoid distractions from the main >>>> design, >>>> > > but I >>>> > > > > > plan >>>> > > > > > > to >>>> > > > > > > > > add >>>> > > > > > > > > >> > > > another background thread to clean up all the >>>> stale >>>> > > > > messages >>>> > > > > > > by >>>> > > > > > > > > >> > comparing >>>> > > > > > > > > >> > > > the message's endOffset with the topic >>>> partition's >>>> > log >>>> > > > > start >>>> > > > > > > > > >> offset. I >>>> > > > > > > > > >> > > > believe this would help remove all the dangling >>>> > > > messages. >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > Thanks, >>>> > > > > > > > > >> > > > Lijun TOng >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > Kamal Chandraprakash < >>>> > [email protected]> >>>> > > > > > > > > 于2026年3月29日周日 >>>> > > > > > > > > >> > > > 22:48写道： >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > Hi Lijun, >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > Sorry for the late reply. Went over the KIP >>>> again. >>>> > > > > Overall >>>> > > > > > > > LGTM. >>>> > > > > > > > > >> Few >>>> > > > > > > > > >> > > > > points: >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > > This KIP aims to solve this issue through >>>> > > > introducing >>>> > > > > > > > another >>>> > > > > > > > > >> > > compacted >>>> > > > > > > > > >> > > > > topic for the brokers to bootstrap the state >>>> from >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > 1. Shall we update the motivation section to >>>> > mention >>>> > > > > that >>>> > > > > > > > > another >>>> > > > > > > > > >> > topic >>>> > > > > > > > > >> > > > is >>>> > > > > > > > > >> > > > > not introduced? >>>> > > > > > > > > >> > > > > 2. Assume that the topic is enabled with >>>> > compaction >>>> > > > and >>>> > > > > > only >>>> > > > > > > > one >>>> > > > > > > > > >> > event >>>> > > > > > > > > >> > > is >>>> > > > > > > > > >> > > > > maintained per segment. If there is a >>>> transient >>>> > > error >>>> > > > in >>>> > > > > > the >>>> > > > > > > > > >> remote >>>> > > > > > > > > >> > log >>>> > > > > > > > > >> > > > > deletion, >>>> > > > > > > > > >> > > > > then the COPY_SEGMENT started / finished >>>> > events >>>> > > > > might >>>> > > > > > be >>>> > > > > > > > > >> > compacted >>>> > > > > > > > > >> > > by >>>> > > > > > > > > >> > > > > the DELETE_SEGMENT_STARTED events. If the >>>> broker >>>> > is >>>> > > > > > > restarted >>>> > > > > > > > > >> during >>>> > > > > > > > > >> > > > > this time, will there be dangling remote >>>> log >>>> > > > > segments? >>>> > > > > > > > > >> Currently, >>>> > > > > > > > > >> > > > > during restart, the broker discards the >>>> events if >>>> > it >>>> > > > > does >>>> > > > > > > not >>>> > > > > > > > > see >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > > COPY_SEGMENT_STARTED events. >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > Thanks, >>>> > > > > > > > > >> > > > > Kamal >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > On Thu, Mar 26, 2026 at 5:08 AM Lijun Tong < >>>> > > > > > > > > >> [email protected]> >>>> > > > > > > > > >> > > > > wrote: >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > > > Hi, >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > I have started a Vote thread for this KIP, >>>> > > > considering >>>> > > > > > all >>>> > > > > > > > > >> > questions >>>> > > > > > > > > >> > > > > raised >>>> > > > > > > > > >> > > > > > so far have been answered. I am happy to >>>> > continue >>>> > > > the >>>> > > > > > > > > >> discussion if >>>> > > > > > > > > >> > > > > needed, >>>> > > > > > > > > >> > > > > > otherwise, this is a friendly reminder on >>>> the >>>> > vote >>>> > > > for >>>> > > > > > > this >>>> > > > > > > > > KIP. >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > Thanks, >>>> > > > > > > > > >> > > > > > Lijun Tong >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > Lijun Tong <[email protected]> >>>> > > 于2026年1月19日周一 >>>> > > > > > > 17:59写道： >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > > > Hey Kamal, >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > > Thanks for raising these questions. Here >>>> are >>>> > my >>>> > > > > > > responses >>>> > > > > > > > to >>>> > > > > > > > > >> your >>>> > > > > > > > > >> > > > > > > questions: >>>> > > > > > > > > >> > > > > > > Q1 and Q2: >>>> > > > > > > > > >> > > > > > > I think both questions boil down to how >>>> to >>>> > > release >>>> > > > > > this >>>> > > > > > > > new >>>> > > > > > > > > >> > > feature, >>>> > > > > > > > > >> > > > > both >>>> > > > > > > > > >> > > > > > > questions are valid concerns. The >>>> solution I >>>> > > have >>>> > > > in >>>> > > > > > > mind >>>> > > > > > > > is >>>> > > > > > > > > >> this >>>> > > > > > > > > >> > > > > feature >>>> > > > > > > > > >> > > > > > > is *gated by the metadata version*. The >>>> new >>>> > > > > tombstone >>>> > > > > > > > > >> semantics >>>> > > > > > > > > >> > and >>>> > > > > > > > > >> > > > the >>>> > > > > > > > > >> > > > > > > additional fields (for example in >>>> > > > > > > > > >> RemoteLogSegmentUpdateRecord) >>>> > > > > > > > > >> > are >>>> > > > > > > > > >> > > > > only >>>> > > > > > > > > >> > > > > > > enabled once the cluster metadata >>>> version is >>>> > > > > upgraded >>>> > > > > > to >>>> > > > > > > > the >>>> > > > > > > > > >> > > version >>>> > > > > > > > > >> > > > > that >>>> > > > > > > > > >> > > > > > > introduces this feature. As long as the >>>> > cluster >>>> > > > > > metadata >>>> > > > > > > > > >> version >>>> > > > > > > > > >> > is >>>> > > > > > > > > >> > > > not >>>> > > > > > > > > >> > > > > > > bumped, the system will not produce >>>> tombstone >>>> > > > > records. >>>> > > > > > > > > >> Therefore, >>>> > > > > > > > > >> > > > > during >>>> > > > > > > > > >> > > > > > > rolling upgrades (mixed 4.2/4.3 >>>> brokers), the >>>> > > > > feature >>>> > > > > > > > > remains >>>> > > > > > > > > >> > > > > effectively >>>> > > > > > > > > >> > > > > > > disabled. Tombstones will only start >>>> being >>>> > > > produced >>>> > > > > > > after >>>> > > > > > > > > the >>>> > > > > > > > > >> > > > metadata >>>> > > > > > > > > >> > > > > > > version is upgraded, at which point all >>>> > brokers >>>> > > > are >>>> > > > > > > > already >>>> > > > > > > > > >> > > required >>>> > > > > > > > > >> > > > to >>>> > > > > > > > > >> > > > > > > support the new behavior. >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > > Since Kafka does not support metadata >>>> version >>>> > > > > > downgrades >>>> > > > > > > > at >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > moment, >>>> > > > > > > > > >> > > > > > > once a metadata version that supports >>>> this >>>> > > feature >>>> > > > > is >>>> > > > > > > > > >> enabled, it >>>> > > > > > > > > >> > > > > cannot >>>> > > > > > > > > >> > > > > > be >>>> > > > > > > > > >> > > > > > > downgraded to a version that does not >>>> support >>>> > > it. >>>> > > > I >>>> > > > > > will >>>> > > > > > > > add >>>> > > > > > > > > >> > these >>>> > > > > > > > > >> > > > > > details >>>> > > > > > > > > >> > > > > > > to the KIP later. >>>> > > > > > > > > >> > > > > > > Q3. This is an *editing mistake* in the >>>> KIP. >>>> > > > Thanks >>>> > > > > > for >>>> > > > > > > > > >> pointing >>>> > > > > > > > > >> > it >>>> > > > > > > > > >> > > > > out — >>>> > > > > > > > > >> > > > > > > the enum value has already been >>>> corrected in >>>> > the >>>> > > > > > latest >>>> > > > > > > > > >> revision >>>> > > > > > > > > >> > to >>>> > > > > > > > > >> > > > > > remove >>>> > > > > > > > > >> > > > > > > the unused placeholder and keep the state >>>> > values >>>> > > > > > > > contiguous >>>> > > > > > > > > >> and >>>> > > > > > > > > >> > > > > > consistent. >>>> > > > > > > > > >> > > > > > > Q4. I don't foresee the quota mechanism >>>> will >>>> > > > > interfere >>>> > > > > > > > with >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > state >>>> > > > > > > > > >> > > > > > > transition in any way so far, let me >>>> know if >>>> > any >>>> > > > > > concern >>>> > > > > > > > > hits >>>> > > > > > > > > >> > you. >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > > Thanks, >>>> > > > > > > > > >> > > > > > > Lijun >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > > Kamal Chandraprakash < >>>> > > > > [email protected]> >>>> > > > > > > > > >> > > 于2026年1月18日周日 >>>> > > > > > > > > >> > > > > > > 00:40写道： >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> Hi Lijun, >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> Thanks for updating the KIP! >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> The updated migration plan looks clean >>>> to me. >>>> > > Few >>>> > > > > > > > > questions: >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> 1. The ConsumerTask in 4.2 Kafka build >>>> does >>>> > not >>>> > > > > > handle >>>> > > > > > > > the >>>> > > > > > > > > >> > > tombstone >>>> > > > > > > > > >> > > > > > >> records. Should the tombstone records >>>> be sent >>>> > > > only >>>> > > > > > when >>>> > > > > > > > all >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > > brokers >>>> > > > > > > > > >> > > > > > >> are >>>> > > > > > > > > >> > > > > > >> upgraded to 4.3 version? >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> 2. Once all the brokers are upgraded >>>> and the >>>> > > > > > > > > >> > __remote_log_metadata >>>> > > > > > > > > >> > > > > topic >>>> > > > > > > > > >> > > > > > >> cleanup policy changed to compact. Then, >>>> > > > > downgrading >>>> > > > > > > the >>>> > > > > > > > > >> brokers >>>> > > > > > > > > >> > > is >>>> > > > > > > > > >> > > > > not >>>> > > > > > > > > >> > > > > > >> allowed as the records without key will >>>> throw >>>> > > an >>>> > > > > > error >>>> > > > > > > > > while >>>> > > > > > > > > >> > > > producing >>>> > > > > > > > > >> > > > > > the >>>> > > > > > > > > >> > > > > > >> compacted topic. Shall we mention this >>>> in the >>>> > > > > > > > compatibility >>>> > > > > > > > > >> > > section? >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> 3. In the RemoteLogSegmentState Enum, >>>> why is >>>> > > the >>>> > > > > > value >>>> > > > > > > 1 >>>> > > > > > > > > >> marked >>>> > > > > > > > > >> > as >>>> > > > > > > > > >> > > > > > unused? >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> 4. Regarding the key >>>> > > > > > > > > >> > > (TopicIdPartition:EndOffset:BrokerLeaderEpoch), >>>> > > > > > > > > >> > > > > we >>>> > > > > > > > > >> > > > > > >> may >>>> > > > > > > > > >> > > > > > >> have to check for scenarios where there >>>> is >>>> > > > segment >>>> > > > > > lag >>>> > > > > > > > due >>>> > > > > > > > > to >>>> > > > > > > > > >> > > remote >>>> > > > > > > > > >> > > > > log >>>> > > > > > > > > >> > > > > > >> write quota. Will the state transition >>>> work >>>> > > > > > correctly? >>>> > > > > > > > Will >>>> > > > > > > > > >> come >>>> > > > > > > > > >> > > > back >>>> > > > > > > > > >> > > > > to >>>> > > > > > > > > >> > > > > > >> this later. >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> Thanks, >>>> > > > > > > > > >> > > > > > >> Kamal >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> On Fri, Jan 16, 2026 at 4:50 AM jian fu >>>> < >>>> > > > > > > > > >> [email protected]> >>>> > > > > > > > > >> > > > wrote: >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >> > Hi Lijun and Kamal >>>> > > > > > > > > >> > > > > > >> > I also think we don't need to keep >>>> > > > delJIanpolicy >>>> > > > > in >>>> > > > > > > > final >>>> > > > > > > > > >> > > > config，if >>>> > > > > > > > > >> > > > > > >> so，we >>>> > > > > > > > > >> > > > > > >> > should always keep remembering all of >>>> our >>>> > > topic >>>> > > > > > > > retention >>>> > > > > > > > > >> time >>>> > > > > > > > > >> > > > must >>>> > > > > > > > > >> > > > > > less >>>> > > > > > > > > >> > > > > > >> > than the value，right？It is one >>>> protect with >>>> > > > risk >>>> > > > > > > > > involved. >>>> > > > > > > > > >> > > > > > >> > Regards >>>> > > > > > > > > >> > > > > > >> > JIan >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> > Lijun Tong <[email protected] >>>> > > > >于2026年1月16日 >>>> > > > > > > > > 周五06:45写道： >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> > > Hey Kamal, >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > Some additional points about the Q4, >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > > The user can decide when to change >>>> > their >>>> > > > > > internal >>>> > > > > > > > > topic >>>> > > > > > > > > >> > > > cleanup >>>> > > > > > > > > >> > > > > > >> policy >>>> > > > > > > > > >> > > > > > >> > to >>>> > > > > > > > > >> > > > > > >> > > > compact. If someone retains >>>> > > > > > > > > >> > > > > > >> > > > the data in the remote storage >>>> for 3 >>>> > > > months, >>>> > > > > > then >>>> > > > > > > > > they >>>> > > > > > > > > >> can >>>> > > > > > > > > >> > > > > migrate >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > the >>>> > > > > > > > > >> > > > > > >> > > > compacted topic after 3 months >>>> > > > > > > > > >> > > > > > >> > > > post rolling out this change. And, >>>> > update >>>> > > > > their >>>> > > > > > > > > cleanup >>>> > > > > > > > > >> > > policy >>>> > > > > > > > > >> > > > > to >>>> > > > > > > > > >> > > > > > >> > > [compact, >>>> > > > > > > > > >> > > > > > >> > > > delete]. >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > I don't think it's a good idea to >>>> keep >>>> > > delete >>>> > > > > in >>>> > > > > > > the >>>> > > > > > > > > >> final >>>> > > > > > > > > >> > > > cleanup >>>> > > > > > > > > >> > > > > > >> policy >>>> > > > > > > > > >> > > > > > >> > > for the topic >>>> `__remote_log_metadata`, as >>>> > > > this >>>> > > > > > > still >>>> > > > > > > > > >> > requires >>>> > > > > > > > > >> > > > the >>>> > > > > > > > > >> > > > > > >> user to >>>> > > > > > > > > >> > > > > > >> > > keep track of the max retention >>>> hours of >>>> > > > topics >>>> > > > > > > that >>>> > > > > > > > > have >>>> > > > > > > > > >> > > remote >>>> > > > > > > > > >> > > > > > >> storage >>>> > > > > > > > > >> > > > > > >> > > enabled, and it's operational >>>> burden. >>>> > It's >>>> > > > also >>>> > > > > > > hard >>>> > > > > > > > to >>>> > > > > > > > > >> > reason >>>> > > > > > > > > >> > > > > about >>>> > > > > > > > > >> > > > > > >> what >>>> > > > > > > > > >> > > > > > >> > > will happen if the user configures >>>> the >>>> > > wrong >>>> > > > > > > > > >> retention.ms. >>>> > > > > > > > > >> > I >>>> > > > > > > > > >> > > > hope >>>> > > > > > > > > >> > > > > > >> this >>>> > > > > > > > > >> > > > > > >> > > makes sense. >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > Thanks, >>>> > > > > > > > > >> > > > > > >> > > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > Lijun Tong <[email protected] >>>> > >>>> > > > > > 于2026年1月15日周四 >>>> > > > > > > > > >> 11:43写道： >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > > > Hey Kamal, >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > Thanks for your reply! I am glad >>>> we are >>>> > > on >>>> > > > > the >>>> > > > > > > same >>>> > > > > > > > > >> page >>>> > > > > > > > > >> > > with >>>> > > > > > > > > >> > > > > > making >>>> > > > > > > > > >> > > > > > >> > the >>>> > > > > > > > > >> > > > > > >> > > > __remote_log_metadata topic >>>> compacted >>>> > > > > optional >>>> > > > > > > for >>>> > > > > > > > > the >>>> > > > > > > > > >> > user >>>> > > > > > > > > >> > > > > now, I >>>> > > > > > > > > >> > > > > > >> will >>>> > > > > > > > > >> > > > > > >> > > > update the KIP with this change. >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > For the Q2: >>>> > > > > > > > > >> > > > > > >> > > > With the key designed as >>>> > > > > > > > > >> > > > > > >> >>>> > TopicId:Partition:EndOffset:BrokerLeaderEpoch, >>>> > > > > > > > > >> > > > > > >> > > > even the same broker retries the >>>> upload >>>> > > > > > multiple >>>> > > > > > > > > times >>>> > > > > > > > > >> for >>>> > > > > > > > > >> > > the >>>> > > > > > > > > >> > > > > > same >>>> > > > > > > > > >> > > > > > >> log >>>> > > > > > > > > >> > > > > > >> > > > segment, the latest retry attempt >>>> with >>>> > > the >>>> > > > > > latest >>>> > > > > > > > > >> segment >>>> > > > > > > > > >> > > UUID >>>> > > > > > > > > >> > > > > > will >>>> > > > > > > > > >> > > > > > >> > > > overwrite the previous attempts' >>>> value >>>> > > > since >>>> > > > > > they >>>> > > > > > > > > share >>>> > > > > > > > > >> > the >>>> > > > > > > > > >> > > > same >>>> > > > > > > > > >> > > > > > >> key, >>>> > > > > > > > > >> > > > > > >> > so >>>> > > > > > > > > >> > > > > > >> > > we >>>> > > > > > > > > >> > > > > > >> > > > don't need to explicitly track the >>>> > failed >>>> > > > > > upload >>>> > > > > > > > > >> metadata, >>>> > > > > > > > > >> > > > > because >>>> > > > > > > > > >> > > > > > >> it's >>>> > > > > > > > > >> > > > > > >> > > > gone already by the later attempt. >>>> > That's >>>> > > > my >>>> > > > > > > > > >> understanding >>>> > > > > > > > > >> > > > about >>>> > > > > > > > > >> > > > > > the >>>> > > > > > > > > >> > > > > > >> > > > RLMCopyTask, correct me if I am >>>> wrong. >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > Thanks, >>>> > > > > > > > > >> > > > > > >> > > > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > > Kamal Chandraprakash < >>>> > > > > > > > [email protected] >>>> > > > > > > > > > >>>> > > > > > > > > >> > > > > > 于2026年1月14日周三 >>>> > > > > > > > > >> > > > > > >> > > > 21:18写道： >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> Hi Lijun, >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Thanks for the reply! >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Q1: Sounds good. Could you >>>> clarify it >>>> > in >>>> > > > the >>>> > > > > > KIP >>>> > > > > > > > > that >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > same >>>> > > > > > > > > >> > > > > > >> > > partitioner >>>> > > > > > > > > >> > > > > > >> > > >> will be used? >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Q2: With >>>> > > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch >>>> > > > > > > > > >> > key, >>>> > > > > > > > > >> > > > if >>>> > > > > > > > > >> > > > > > the >>>> > > > > > > > > >> > > > > > >> > same >>>> > > > > > > > > >> > > > > > >> > > >> broker retries the upload due to >>>> > > > > intermittent >>>> > > > > > > > > >> > > > > > >> > > >> issues in object storage (or) >>>> RLMM. >>>> > > Then, >>>> > > > > > those >>>> > > > > > > > > failed >>>> > > > > > > > > >> > > upload >>>> > > > > > > > > >> > > > > > >> metadata >>>> > > > > > > > > >> > > > > > >> > > >> also >>>> > > > > > > > > >> > > > > > >> > > >> need to be cleared. >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Q3: We may have to skip the null >>>> value >>>> > > > > records >>>> > > > > > > in >>>> > > > > > > > > the >>>> > > > > > > > > >> > > > > > ConsumerTask. >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Q4a: The idea is to keep the >>>> cleanup >>>> > > > policy >>>> > > > > as >>>> > > > > > > > > >> "delete" >>>> > > > > > > > > >> > and >>>> > > > > > > > > >> > > > > also >>>> > > > > > > > > >> > > > > > >> send >>>> > > > > > > > > >> > > > > > >> > > the >>>> > > > > > > > > >> > > > > > >> > > >> tombstone markers >>>> > > > > > > > > >> > > > > > >> > > >> to the existing >>>> > `__remote_log_metadata` >>>> > > > > topic. >>>> > > > > > > > And, >>>> > > > > > > > > >> > handle >>>> > > > > > > > > >> > > > the >>>> > > > > > > > > >> > > > > > >> > tombstone >>>> > > > > > > > > >> > > > > > >> > > >> records in the ConsumerTask. >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> The user can decide when to >>>> change >>>> > their >>>> > > > > > > internal >>>> > > > > > > > > >> topic >>>> > > > > > > > > >> > > > cleanup >>>> > > > > > > > > >> > > > > > >> policy >>>> > > > > > > > > >> > > > > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> compact. If someone retains >>>> > > > > > > > > >> > > > > > >> > > >> the data in the remote storage >>>> for 3 >>>> > > > months, >>>> > > > > > > then >>>> > > > > > > > > they >>>> > > > > > > > > >> > can >>>> > > > > > > > > >> > > > > > migrate >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > > the >>>> > > > > > > > > >> > > > > > >> > > >> compacted topic after 3 months >>>> > > > > > > > > >> > > > > > >> > > >> post rolling out this change. >>>> And, >>>> > > update >>>> > > > > > their >>>> > > > > > > > > >> cleanup >>>> > > > > > > > > >> > > > policy >>>> > > > > > > > > >> > > > > to >>>> > > > > > > > > >> > > > > > >> > > >> [compact, >>>> > > > > > > > > >> > > > > > >> > > >> delete]. >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> Thanks, >>>> > > > > > > > > >> > > > > > >> > > >> Kamal >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> On Thu, Jan 15, 2026 at 4:12 AM >>>> Lijun >>>> > > > Tong < >>>> > > > > > > > > >> > > > > > >> [email protected]> >>>> > > > > > > > > >> > > > > > >> > > >> wrote: >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >> > Hey Jian, >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > Thanks for your time to review >>>> this >>>> > > > KIP. I >>>> > > > > > > > > >> appreciate >>>> > > > > > > > > >> > > that >>>> > > > > > > > > >> > > > > you >>>> > > > > > > > > >> > > > > > >> > > propose a >>>> > > > > > > > > >> > > > > > >> > > >> > simpler migration solution to >>>> > onboard >>>> > > > the >>>> > > > > > new >>>> > > > > > > > > >> feature. >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > There are 2 points that I >>>> think can >>>> > be >>>> > > > > > further >>>> > > > > > > > > >> refined >>>> > > > > > > > > >> > > on: >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > 1). make the topic compacted >>>> > optional, >>>> > > > > > > although >>>> > > > > > > > > the >>>> > > > > > > > > >> new >>>> > > > > > > > > >> > > > > feature >>>> > > > > > > > > >> > > > > > >> will >>>> > > > > > > > > >> > > > > > >> > > >> > continue to emit tombstone >>>> message >>>> > for >>>> > > > > those >>>> > > > > > > > > expired >>>> > > > > > > > > >> > log >>>> > > > > > > > > >> > > > > > segments >>>> > > > > > > > > >> > > > > > >> > even >>>> > > > > > > > > >> > > > > > >> > > >> when >>>> > > > > > > > > >> > > > > > >> > > >> > the topic is still on >>>> time-based >>>> > > > retention >>>> > > > > > > mode, >>>> > > > > > > > > so >>>> > > > > > > > > >> > once >>>> > > > > > > > > >> > > > user >>>> > > > > > > > > >> > > > > > >> > switched >>>> > > > > > > > > >> > > > > > >> > > >> to >>>> > > > > > > > > >> > > > > > >> > > >> > using the compacted topic, >>>> those >>>> > > expired >>>> > > > > > > > messages >>>> > > > > > > > > >> can >>>> > > > > > > > > >> > > still >>>> > > > > > > > > >> > > > > be >>>> > > > > > > > > >> > > > > > >> > deleted >>>> > > > > > > > > >> > > > > > >> > > >> > despite the topic is not >>>> retention >>>> > > based >>>> > > > > > > > anymore. >>>> > > > > > > > > >> > > > > > >> > > >> > 2). we need to expose some >>>> flag to >>>> > the >>>> > > > > user >>>> > > > > > to >>>> > > > > > > > > >> indicate >>>> > > > > > > > > >> > > > > whether >>>> > > > > > > > > >> > > > > > >> the >>>> > > > > > > > > >> > > > > > >> > > >> topic >>>> > > > > > > > > >> > > > > > >> > > >> > can be flipped to compacted by >>>> > > checking >>>> > > > > > > whether >>>> > > > > > > > > all >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > old >>>> > > > > > > > > >> > > > > > >> format >>>> > > > > > > > > >> > > > > > >> > > >> > keyed-less message has >>>> expired, and >>>> > > > allow >>>> > > > > > user >>>> > > > > > > > to >>>> > > > > > > > > >> > choose >>>> > > > > > > > > >> > > to >>>> > > > > > > > > >> > > > > > flip >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > > >> > compacted only when the flag is >>>> > true. >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > Thanks for sharing your idea. >>>> I will >>>> > > > > update >>>> > > > > > > the >>>> > > > > > > > > KIP >>>> > > > > > > > > >> > later >>>> > > > > > > > > >> > > > > with >>>> > > > > > > > > >> > > > > > >> this >>>> > > > > > > > > >> > > > > > >> > > new >>>> > > > > > > > > >> > > > > > >> > > >> > idea. >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > Best, >>>> > > > > > > > > >> > > > > > >> > > >> > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > jian fu <[email protected]> >>>> > > > > > 于2026年1月12日周一 >>>> > > > > > > > > >> 04:55写道： >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> > > Hi Lijun Tong: >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > Thanks for your KIP which >>>> raise >>>> > this >>>> > > > > > > critical >>>> > > > > > > > > >> issue. >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > what about just keep one >>>> topic >>>> > > instead >>>> > > > > of >>>> > > > > > > > > involve >>>> > > > > > > > > >> > > another >>>> > > > > > > > > >> > > > > > >> topic. >>>> > > > > > > > > >> > > > > > >> > > >> > > for existed topic data's >>>> > migration. >>>> > > > > maybe >>>> > > > > > we >>>> > > > > > > > can >>>> > > > > > > > > >> use >>>> > > > > > > > > >> > > this >>>> > > > > > > > > >> > > > > way >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > > solve >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > issue: >>>> > > > > > > > > >> > > > > > >> > > >> > > (1) set the retention date > >>>> all >>>> > of >>>> > > > > topic >>>> > > > > > > > which >>>> > > > > > > > > >> > enable >>>> > > > > > > > > >> > > > > remote >>>> > > > > > > > > >> > > > > > >> > > >> storage's >>>> > > > > > > > > >> > > > > > >> > > >> > > retention time >>>> > > > > > > > > >> > > > > > >> > > >> > > (2) deploy new kafka version >>>> with >>>> > > > > feature: >>>> > > > > > > > > which >>>> > > > > > > > > >> > send >>>> > > > > > > > > >> > > > the >>>> > > > > > > > > >> > > > > > >> message >>>> > > > > > > > > >> > > > > > >> > > >> with >>>> > > > > > > > > >> > > > > > >> > > >> > key >>>> > > > > > > > > >> > > > > > >> > > >> > > (3) wait all the message >>>> expired >>>> > and >>>> > > > new >>>> > > > > > > > message >>>> > > > > > > > > >> with >>>> > > > > > > > > >> > > key >>>> > > > > > > > > >> > > > > > >> coming >>>> > > > > > > > > >> > > > > > >> > to >>>> > > > > > > > > >> > > > > > >> > > >> the >>>> > > > > > > > > >> > > > > > >> > > >> > > topic >>>> > > > > > > > > >> > > > > > >> > > >> > > (4) convert the topic to >>>> compact >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > I don't test it. Just >>>> propose this >>>> > > > > > solution >>>> > > > > > > > > >> according >>>> > > > > > > > > >> > > to >>>> > > > > > > > > >> > > > > code >>>> > > > > > > > > >> > > > > > >> > review >>>> > > > > > > > > >> > > > > > >> > > >> > > result. just for your >>>> reference. >>>> > > > > > > > > >> > > > > > >> > > >> > > The steps maybe a little >>>> complex. >>>> > > but >>>> > > > it >>>> > > > > > can >>>> > > > > > > > > >> avoiding >>>> > > > > > > > > >> > > add >>>> > > > > > > > > >> > > > > new >>>> > > > > > > > > >> > > > > > >> > topic. >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > Regards >>>> > > > > > > > > >> > > > > > >> > > >> > > Jian >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > Lijun Tong < >>>> > [email protected] >>>> > > > >>>> > > > > > > > > 于2026年1月8日周四 >>>> > > > > > > > > >> > > > 09:17写道： >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Hey Kamal, >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Thanks for your time for >>>> the >>>> > > review. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Here is my response to your >>>> > > > questions: >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Q1 At this point, I don’t >>>> see a >>>> > > need >>>> > > > > to >>>> > > > > > > > change >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > RemoteLogMetadataTopicPartitioner >>>> > > > for >>>> > > > > > this >>>> > > > > > > > > >> design. >>>> > > > > > > > > >> > > > > Nothing >>>> > > > > > > > > >> > > > > > in >>>> > > > > > > > > >> > > > > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > current >>>> > > > > > > > > >> > > > > > >> > > >> > > > approach appears to >>>> require a >>>> > > > > > partitioner >>>> > > > > > > > > >> change, >>>> > > > > > > > > >> > but >>>> > > > > > > > > >> > > > I’m >>>> > > > > > > > > >> > > > > > >> open >>>> > > > > > > > > >> > > > > > >> > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > revisiting if a concrete >>>> need >>>> > > > arises. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Q2 I have some reservations >>>> > about >>>> > > > > using >>>> > > > > > > > > >> > > SegmentId:State >>>> > > > > > > > > >> > > > > as >>>> > > > > > > > > >> > > > > > >> the >>>> > > > > > > > > >> > > > > > >> > > key. >>>> > > > > > > > > >> > > > > > >> > > >> A >>>> > > > > > > > > >> > > > > > >> > > >> > > > practical challenge we see >>>> today >>>> > > is >>>> > > > > that >>>> > > > > > > the >>>> > > > > > > > > >> same >>>> > > > > > > > > >> > > > logical >>>> > > > > > > > > >> > > > > > >> > segment >>>> > > > > > > > > >> > > > > > >> > > >> can >>>> > > > > > > > > >> > > > > > >> > > >> > be >>>> > > > > > > > > >> > > > > > >> > > >> > > > retried multiple times with >>>> > > > different >>>> > > > > > > > > SegmentIds >>>> > > > > > > > > >> > > across >>>> > > > > > > > > >> > > > > > >> brokers. >>>> > > > > > > > > >> > > > > > >> > > If >>>> > > > > > > > > >> > > > > > >> > > >> the >>>> > > > > > > > > >> > > > > > >> > > >> > > key >>>> > > > > > > > > >> > > > > > >> > > >> > > > is SegmentId-based, it >>>> becomes >>>> > > > harder >>>> > > > > to >>>> > > > > > > > > >> discover >>>> > > > > > > > > >> > and >>>> > > > > > > > > >> > > > > > >> tombstone >>>> > > > > > > > > >> > > > > > >> > > all >>>> > > > > > > > > >> > > > > > >> > > >> > > related >>>> > > > > > > > > >> > > > > > >> > > >> > > > attempts when the segment >>>> > > eventually >>>> > > > > > > > expires. >>>> > > > > > > > > >> The >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch >>>> > > > > > > > > >> key >>>> > > > > > > > > >> > is >>>> > > > > > > > > >> > > > > > >> > deterministic >>>> > > > > > > > > >> > > > > > >> > > >> for >>>> > > > > > > > > >> > > > > > >> > > >> > a >>>> > > > > > > > > >> > > > > > >> > > >> > > > logical segment attempt and >>>> > helps >>>> > > > > group >>>> > > > > > > > > retries >>>> > > > > > > > > >> by >>>> > > > > > > > > >> > > > epoch, >>>> > > > > > > > > >> > > > > > >> which >>>> > > > > > > > > >> > > > > > >> > > >> > > simplifies >>>> > > > > > > > > >> > > > > > >> > > >> > > > cleanup and reasoning about >>>> > state. >>>> > > > I’d >>>> > > > > > > love >>>> > > > > > > > to >>>> > > > > > > > > >> > > > understand >>>> > > > > > > > > >> > > > > > the >>>> > > > > > > > > >> > > > > > >> > > >> benefits >>>> > > > > > > > > >> > > > > > >> > > >> > > > you’re seeing with >>>> > SegmentId:State >>>> > > > > > > compared >>>> > > > > > > > to >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > > > >> > > >> offset/epoch-based >>>> > > > > > > > > >> > > > > > >> > > >> > key >>>> > > > > > > > > >> > > > > > >> > > >> > > > so we can weigh the >>>> trade-offs. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > On partitioning: with this >>>> > > proposal, >>>> > > > > all >>>> > > > > > > > > states >>>> > > > > > > > > >> > for a >>>> > > > > > > > > >> > > > > given >>>> > > > > > > > > >> > > > > > >> user >>>> > > > > > > > > >> > > > > > >> > > >> > > > topic-partition still map >>>> to the >>>> > > > same >>>> > > > > > > > metadata >>>> > > > > > > > > >> > > > partition. >>>> > > > > > > > > >> > > > > > >> That >>>> > > > > > > > > >> > > > > > >> > > >> remains >>>> > > > > > > > > >> > > > > > >> > > >> > > true >>>> > > > > > > > > >> > > > > > >> > > >> > > > for the existing >>>> > > > __remote_log_metadata >>>> > > > > > > > > >> (unchanged >>>> > > > > > > > > >> > > > > > >> partitioner) >>>> > > > > > > > > >> > > > > > >> > and >>>> > > > > > > > > >> > > > > > >> > > >> for >>>> > > > > > > > > >> > > > > > >> > > >> > > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > new >>>> > > __remote_log_metadata_compacted, >>>> > > > > > > > > preserving >>>> > > > > > > > > >> the >>>> > > > > > > > > >> > > > > > >> properties >>>> > > > > > > > > >> > > > > > >> > > >> > > > RemoteMetadataCache relies >>>> on. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Q3 It should be fine for >>>> > > > ConsumerTask >>>> > > > > to >>>> > > > > > > > > ignore >>>> > > > > > > > > >> > > > tombstone >>>> > > > > > > > > >> > > > > > >> > records >>>> > > > > > > > > >> > > > > > >> > > >> (null >>>> > > > > > > > > >> > > > > > >> > > >> > > > values) and no-op. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Q4 Although TBRLMM is a >>>> sample >>>> > > RLMM >>>> > > > > > > > > >> implementation, >>>> > > > > > > > > >> > > > it’s >>>> > > > > > > > > >> > > > > > >> > currently >>>> > > > > > > > > >> > > > > > >> > > >> the >>>> > > > > > > > > >> > > > > > >> > > >> > > only >>>> > > > > > > > > >> > > > > > >> > > >> > > > OSS option and is widely >>>> used. >>>> > The >>>> > > > new >>>> > > > > > > > > >> > > > > > >> > > >> __remote_log_metadata_compacted >>>> > > > > > > > > >> > > > > > >> > > >> > > > topic offers clear >>>> operational >>>> > > > > benefits >>>> > > > > > in >>>> > > > > > > > > that >>>> > > > > > > > > >> > > > context. >>>> > > > > > > > > >> > > > > We >>>> > > > > > > > > >> > > > > > >> can >>>> > > > > > > > > >> > > > > > >> > > also >>>> > > > > > > > > >> > > > > > >> > > >> > > > provide a configuration to >>>> let >>>> > > users >>>> > > > > > > choose >>>> > > > > > > > > >> whether >>>> > > > > > > > > >> > > > they >>>> > > > > > > > > >> > > > > > >> want to >>>> > > > > > > > > >> > > > > > >> > > >> keep >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > audit topic >>>> > > (__remote_log_metadata) >>>> > > > in >>>> > > > > > > their >>>> > > > > > > > > >> > cluster. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Q4a Enabling compaction on >>>> > > > > > > > > __remote_log_metadata >>>> > > > > > > > > >> > > alone >>>> > > > > > > > > >> > > > > may >>>> > > > > > > > > >> > > > > > >> not >>>> > > > > > > > > >> > > > > > >> > > fully >>>> > > > > > > > > >> > > > > > >> > > >> > > > address the unbounded >>>> growth, >>>> > > since >>>> > > > we >>>> > > > > > > also >>>> > > > > > > > > >> need to >>>> > > > > > > > > >> > > > emit >>>> > > > > > > > > >> > > > > > >> > > tombstones >>>> > > > > > > > > >> > > > > > >> > > >> for >>>> > > > > > > > > >> > > > > > >> > > >> > > > expired keys to delete >>>> them. >>>> > > > Deferring >>>> > > > > > > > > >> compaction >>>> > > > > > > > > >> > and >>>> > > > > > > > > >> > > > > > >> > tombstoning >>>> > > > > > > > > >> > > > > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > user >>>> > > > > > > > > >> > > > > > >> > > >> > > > configuration could make >>>> the >>>> > code >>>> > > > flow >>>> > > > > > > > > >> complicated, >>>> > > > > > > > > >> > > > also >>>> > > > > > > > > >> > > > > > add >>>> > > > > > > > > >> > > > > > >> > > >> > operational >>>> > > > > > > > > >> > > > > > >> > > >> > > > complexity and make >>>> outcomes >>>> > less >>>> > > > > > > > predictable. >>>> > > > > > > > > >> The >>>> > > > > > > > > >> > > > > proposal >>>> > > > > > > > > >> > > > > > >> aims >>>> > > > > > > > > >> > > > > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > provide >>>> > > > > > > > > >> > > > > > >> > > >> > > > a consistent experience by >>>> > > defining >>>> > > > > > > > > >> deterministic >>>> > > > > > > > > >> > > keys >>>> > > > > > > > > >> > > > > and >>>> > > > > > > > > >> > > > > > >> > > emitting >>>> > > > > > > > > >> > > > > > >> > > >> > > > tombstones as part of the >>>> > broker’s >>>> > > > > > > > > >> > responsibilities, >>>> > > > > > > > > >> > > > > while >>>> > > > > > > > > >> > > > > > >> still >>>> > > > > > > > > >> > > > > > >> > > >> > allowing >>>> > > > > > > > > >> > > > > > >> > > >> > > > users to opt out of the >>>> audit >>>> > > topic >>>> > > > if >>>> > > > > > > they >>>> > > > > > > > > >> prefer. >>>> > > > > > > > > >> > > > But I >>>> > > > > > > > > >> > > > > > am >>>> > > > > > > > > >> > > > > > >> > open >>>> > > > > > > > > >> > > > > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > more >>>> > > > > > > > > >> > > > > > >> > > >> > > > discussion if there is any >>>> > > concrete >>>> > > > > > need I >>>> > > > > > > > > don't >>>> > > > > > > > > >> > > > foresee. >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Thanks, >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > Kamal Chandraprakash < >>>> > > > > > > > > >> > [email protected] >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > 于2026年1月6日周二 >>>> > > > > > > > > >> > > > > > >> > > >> > > > 01:01写道： >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > Hi Lijun, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > Thanks for the KIP! Went >>>> over >>>> > > the >>>> > > > > > first >>>> > > > > > > > > pass. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > Few Questions: >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > 1. Are we going to >>>> maintain >>>> > the >>>> > > > same >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> RemoteLogMetadataTopicPartitioner >>>> > > > > > > > > >> > > > > > >> > > >> > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataTopicPartitioner.java >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > for both the topics? It >>>> is not >>>> > > > clear >>>> > > > > > in >>>> > > > > > > > the >>>> > > > > > > > > >> KIP, >>>> > > > > > > > > >> > > > could >>>> > > > > > > > > >> > > > > > you >>>> > > > > > > > > >> > > > > > >> > > clarify >>>> > > > > > > > > >> > > > > > >> > > >> > it? >>>> > > > > > > > > >> > > > > > >> > > >> > > > > 2. Can the key be >>>> changed to >>>> > > > > > > > SegmentId:State >>>> > > > > > > > > >> > > instead >>>> > > > > > > > > >> > > > of >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch >>>> > > > > > > > > >> if >>>> > > > > > > > > >> > > the >>>> > > > > > > > > >> > > > > same >>>> > > > > > > > > >> > > > > > >> > > >> partitioner >>>> > > > > > > > > >> > > > > > >> > > >> > > is >>>> > > > > > > > > >> > > > > > >> > > >> > > > > used? It is good to >>>> maintain >>>> > all >>>> > > > the >>>> > > > > > > > segment >>>> > > > > > > > > >> > states >>>> > > > > > > > > >> > > > > for a >>>> > > > > > > > > >> > > > > > >> > > >> > > > > user-topic-partition in >>>> the >>>> > same >>>> > > > > > > metadata >>>> > > > > > > > > >> > > partition. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > 3. Should we have to >>>> handle >>>> > the >>>> > > > > > records >>>> > > > > > > > with >>>> > > > > > > > > >> null >>>> > > > > > > > > >> > > > value >>>> > > > > > > > > >> > > > > > >> > > >> (tombstone) >>>> > > > > > > > > >> > > > > > >> > > >> > in >>>> > > > > > > > > >> > > > > > >> > > >> > > > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > > ConsumerTask >>>> > > > > > > > > >> > > > > > >> > > >> > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/ConsumerTask.java?L166 >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > ? >>>> > > > > > > > > >> > > > > > >> > > >> > > > > 4. TBRLMM >>>> > > > > > > > > >> > > > > > >> > > >> > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > is a sample plugin >>>> > > implementation >>>> > > > of >>>> > > > > > > RLMM. >>>> > > > > > > > > Not >>>> > > > > > > > > >> > sure >>>> > > > > > > > > >> > > > > > whether >>>> > > > > > > > > >> > > > > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > community >>>> > > > > > > > > >> > > > > > >> > > >> > > > > will agree to add one >>>> more >>>> > > > internal >>>> > > > > > > topic >>>> > > > > > > > > for >>>> > > > > > > > > >> > this >>>> > > > > > > > > >> > > > > plugin >>>> > > > > > > > > >> > > > > > >> > impl. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > 4a. Can we modify the new >>>> > > messages >>>> > > > > to >>>> > > > > > > the >>>> > > > > > > > > >> > > > > > >> > __remote_log_metadata >>>> > > > > > > > > >> > > > > > >> > > >> topic >>>> > > > > > > > > >> > > > > > >> > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > contain the key and >>>> leave it >>>> > to >>>> > > > the >>>> > > > > > user >>>> > > > > > > > to >>>> > > > > > > > > >> > enable >>>> > > > > > > > > >> > > > > > >> compaction >>>> > > > > > > > > >> > > > > > >> > > for >>>> > > > > > > > > >> > > > > > >> > > >> > this >>>> > > > > > > > > >> > > > > > >> > > >> > > > > topic if they need? >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > Thanks, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > Kamal >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > On Tue, Jan 6, 2026 at >>>> 7:35 AM >>>> > > > Lijun >>>> > > > > > > Tong >>>> > > > > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> [email protected]> >>>> > > > > > > > > >> > > > > > >> > > >> > > > wrote: >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Hey Henry, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Thank you for your >>>> time and >>>> > > > > > response! >>>> > > > > > > I >>>> > > > > > > > > >> really >>>> > > > > > > > > >> > > like >>>> > > > > > > > > >> > > > > > your >>>> > > > > > > > > >> > > > > > >> > > >> KIP-1248 >>>> > > > > > > > > >> > > > > > >> > > >> > > about >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > offloading the >>>> consumption >>>> > of >>>> > > > > remote >>>> > > > > > > log >>>> > > > > > > > > >> away >>>> > > > > > > > > >> > > from >>>> > > > > > > > > >> > > > > the >>>> > > > > > > > > >> > > > > > >> > broker, >>>> > > > > > > > > >> > > > > > >> > > >> and >>>> > > > > > > > > >> > > > > > >> > > >> > I >>>> > > > > > > > > >> > > > > > >> > > >> > > > > think >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > with that change, the >>>> topic >>>> > > that >>>> > > > > > > enables >>>> > > > > > > > > the >>>> > > > > > > > > >> > > tiered >>>> > > > > > > > > >> > > > > > >> storage >>>> > > > > > > > > >> > > > > > >> > > can >>>> > > > > > > > > >> > > > > > >> > > >> > also >>>> > > > > > > > > >> > > > > > >> > > >> > > > have >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > longer retention >>>> > > configurations >>>> > > > > and >>>> > > > > > > > would >>>> > > > > > > > > >> > benefit >>>> > > > > > > > > >> > > > > from >>>> > > > > > > > > >> > > > > > >> this >>>> > > > > > > > > >> > > > > > >> > > KIP >>>> > > > > > > > > >> > > > > > >> > > >> > too. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Some suggestions: In >>>> your >>>> > > > example >>>> > > > > > > > > >> scenarios, it >>>> > > > > > > > > >> > > > would >>>> > > > > > > > > >> > > > > > >> also >>>> > > > > > > > > >> > > > > > >> > be >>>> > > > > > > > > >> > > > > > >> > > >> good >>>> > > > > > > > > >> > > > > > >> > > >> > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > add >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > an example of remote >>>> log >>>> > > > segment >>>> > > > > > > > > deletion >>>> > > > > > > > > >> > > > triggered >>>> > > > > > > > > >> > > > > > by >>>> > > > > > > > > >> > > > > > >> > > >> retention >>>> > > > > > > > > >> > > > > > >> > > >> > > > policy >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > which will trigger >>>> > > generation >>>> > > > of >>>> > > > > > > > > tombstone >>>> > > > > > > > > >> > > event >>>> > > > > > > > > >> > > > > into >>>> > > > > > > > > >> > > > > > >> > > metadata >>>> > > > > > > > > >> > > > > > >> > > >> > > topic >>>> > > > > > > > > >> > > > > > >> > > >> > > > > and >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > trigger log >>>> > > > compaction/deletion >>>> > > > > 24 >>>> > > > > > > > hour >>>> > > > > > > > > >> > later, >>>> > > > > > > > > >> > > I >>>> > > > > > > > > >> > > > > > think >>>> > > > > > > > > >> > > > > > >> > this >>>> > > > > > > > > >> > > > > > >> > > is >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > key >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > event to cap the >>>> metadata >>>> > > > topic >>>> > > > > > > size. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Regarding to this >>>> > suggestion, >>>> > > I >>>> > > > am >>>> > > > > > not >>>> > > > > > > > > sure >>>> > > > > > > > > >> > > whether >>>> > > > > > > > > >> > > > > > >> > Scenario 4 >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618613#KIP1266:BoundingTheNumberOfRemoteLogMetadataMessagesviaCompactedTopic-Scenario4:SegmentDeletion >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > has >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > covered it. I can add >>>> more >>>> > > rows >>>> > > > in >>>> > > > > > the >>>> > > > > > > > > >> Timeline >>>> > > > > > > > > >> > > > Table >>>> > > > > > > > > >> > > > > > >> like >>>> > > > > > > > > >> > > > > > >> > > >> > T5+24hour >>>> > > > > > > > > >> > > > > > >> > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > indicate the messages >>>> are >>>> > gone >>>> > > > by >>>> > > > > > then >>>> > > > > > > > to >>>> > > > > > > > > >> > > > explicitly >>>> > > > > > > > > >> > > > > > show >>>> > > > > > > > > >> > > > > > >> > that >>>> > > > > > > > > >> > > > > > >> > > >> > > messages >>>> > > > > > > > > >> > > > > > >> > > >> > > > > are >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > deleted, thus the >>>> number of >>>> > > > > messages >>>> > > > > > > are >>>> > > > > > > > > >> capped >>>> > > > > > > > > >> > > in >>>> > > > > > > > > >> > > > > the >>>> > > > > > > > > >> > > > > > >> > topic. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Regarding whether the >>>> topic >>>> > > > > > > > > >> > __remote_log_metadata >>>> > > > > > > > > >> > > > is >>>> > > > > > > > > >> > > > > > >> still >>>> > > > > > > > > >> > > > > > >> > > >> > > necessary, I >>>> > > > > > > > > >> > > > > > >> > > >> > > > > am >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > inclined to continue >>>> to have >>>> > > > this >>>> > > > > > > topic >>>> > > > > > > > at >>>> > > > > > > > > >> > least >>>> > > > > > > > > >> > > > for >>>> > > > > > > > > >> > > > > > >> > debugging >>>> > > > > > > > > >> > > > > > >> > > >> > > purposes >>>> > > > > > > > > >> > > > > > >> > > >> > > > > so >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > we can build confidence >>>> > about >>>> > > > the >>>> > > > > > > > > compacted >>>> > > > > > > > > >> > topic >>>> > > > > > > > > >> > > > > > >> change, we >>>> > > > > > > > > >> > > > > > >> > > can >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > always choose to >>>> remove this >>>> > > > topic >>>> > > > > > in >>>> > > > > > > > the >>>> > > > > > > > > >> > future >>>> > > > > > > > > >> > > > once >>>> > > > > > > > > >> > > > > > we >>>> > > > > > > > > >> > > > > > >> all >>>> > > > > > > > > >> > > > > > >> > > >> agree >>>> > > > > > > > > >> > > > > > >> > > >> > it >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > provides limited value >>>> for >>>> > the >>>> > > > > > users. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Thanks, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > Henry Haiying Cai via >>>> dev < >>>> > > > > > > > > >> > [email protected]> >>>> > > > > > > > > >> > > > > > >> > 于2026年1月5日周一 >>>> > > > > > > > > >> > > > > > >> > > >> > > 16:19写道： >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Lijun, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Thanks for the >>>> proposal >>>> > and >>>> > > I >>>> > > > > > liked >>>> > > > > > > > your >>>> > > > > > > > > >> idea >>>> > > > > > > > > >> > > of >>>> > > > > > > > > >> > > > > > using >>>> > > > > > > > > >> > > > > > >> a >>>> > > > > > > > > >> > > > > > >> > > >> > compacted >>>> > > > > > > > > >> > > > > > >> > > >> > > > > topic >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > for tiered storage >>>> > metadata >>>> > > > > topic. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > In our setup, we >>>> have set >>>> > a >>>> > > > > > shorter >>>> > > > > > > > > >> retention >>>> > > > > > > > > >> > > (3 >>>> > > > > > > > > >> > > > > > days) >>>> > > > > > > > > >> > > > > > >> for >>>> > > > > > > > > >> > > > > > >> > > the >>>> > > > > > > > > >> > > > > > >> > > >> > > tiered >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > storage metadata >>>> topic to >>>> > > > > control >>>> > > > > > > the >>>> > > > > > > > > size >>>> > > > > > > > > >> > > > growth. >>>> > > > > > > > > >> > > > > > We >>>> > > > > > > > > >> > > > > > >> can >>>> > > > > > > > > >> > > > > > >> > > do >>>> > > > > > > > > >> > > > > > >> > > >> > that >>>> > > > > > > > > >> > > > > > >> > > >> > > > > since >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > we >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > control all topic's >>>> > > retention >>>> > > > > > policy >>>> > > > > > > > in >>>> > > > > > > > > >> our >>>> > > > > > > > > >> > > > > clusters >>>> > > > > > > > > >> > > > > > >> and >>>> > > > > > > > > >> > > > > > >> > we >>>> > > > > > > > > >> > > > > > >> > > >> set a >>>> > > > > > > > > >> > > > > > >> > > >> > > > > uniform >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > retention.policy for >>>> all >>>> > our >>>> > > > > > tiered >>>> > > > > > > > > >> storage >>>> > > > > > > > > >> > > > topics. >>>> > > > > > > > > >> > > > > > I >>>> > > > > > > > > >> > > > > > >> can >>>> > > > > > > > > >> > > > > > >> > > see >>>> > > > > > > > > >> > > > > > >> > > >> > > other >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > users/companies will >>>> not >>>> > be >>>> > > > able >>>> > > > > > to >>>> > > > > > > > > >> enforce >>>> > > > > > > > > >> > > that >>>> > > > > > > > > >> > > > > > >> retention >>>> > > > > > > > > >> > > > > > >> > > >> policy >>>> > > > > > > > > >> > > > > > >> > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > all >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > tiered storage >>>> topics. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Some suggestions: In >>>> your >>>> > > > > example >>>> > > > > > > > > >> scenarios, >>>> > > > > > > > > >> > it >>>> > > > > > > > > >> > > > > would >>>> > > > > > > > > >> > > > > > >> also >>>> > > > > > > > > >> > > > > > >> > > be >>>> > > > > > > > > >> > > > > > >> > > >> > good >>>> > > > > > > > > >> > > > > > >> > > >> > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > add >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > an example of remote >>>> log >>>> > > > segment >>>> > > > > > > > > deletion >>>> > > > > > > > > >> > > > triggered >>>> > > > > > > > > >> > > > > > by >>>> > > > > > > > > >> > > > > > >> > > >> retention >>>> > > > > > > > > >> > > > > > >> > > >> > > > policy >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > which will trigger >>>> > > generation >>>> > > > of >>>> > > > > > > > > tombstone >>>> > > > > > > > > >> > > event >>>> > > > > > > > > >> > > > > into >>>> > > > > > > > > >> > > > > > >> > > metadata >>>> > > > > > > > > >> > > > > > >> > > >> > > topic >>>> > > > > > > > > >> > > > > > >> > > >> > > > > and >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > trigger log >>>> > > > compaction/deletion >>>> > > > > 24 >>>> > > > > > > > hour >>>> > > > > > > > > >> > later, >>>> > > > > > > > > >> > > I >>>> > > > > > > > > >> > > > > > think >>>> > > > > > > > > >> > > > > > >> > this >>>> > > > > > > > > >> > > > > > >> > > is >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > key >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > event to cap the >>>> metadata >>>> > > > topic >>>> > > > > > > size. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > For the original >>>> unbounded >>>> > > > > > > > > >> > remote_log_metadata >>>> > > > > > > > > >> > > > > topic, >>>> > > > > > > > > >> > > > > > >> I am >>>> > > > > > > > > >> > > > > > >> > > not >>>> > > > > > > > > >> > > > > > >> > > >> > sure >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > whether we still >>>> need it >>>> > or >>>> > > > not. >>>> > > > > > If >>>> > > > > > > > it >>>> > > > > > > > > is >>>> > > > > > > > > >> > left >>>> > > > > > > > > >> > > > > only >>>> > > > > > > > > >> > > > > > >> for >>>> > > > > > > > > >> > > > > > >> > > audit >>>> > > > > > > > > >> > > > > > >> > > >> > > trail >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > purpose, people can >>>> set >>>> > up a >>>> > > > > data >>>> > > > > > > > > >> ingestion >>>> > > > > > > > > >> > > > > pipeline >>>> > > > > > > > > >> > > > > > to >>>> > > > > > > > > >> > > > > > >> > > ingest >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > content >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > of metadata topic >>>> into a >>>> > > > > separate >>>> > > > > > > > > storage >>>> > > > > > > > > >> > > > location. >>>> > > > > > > > > >> > > > > > I >>>> > > > > > > > > >> > > > > > >> > think >>>> > > > > > > > > >> > > > > > >> > > >> we >>>> > > > > > > > > >> > > > > > >> > > >> > can >>>> > > > > > > > > >> > > > > > >> > > >> > > > > have >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > a >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > flag to have only one >>>> > > metadata >>>> > > > > > topic >>>> > > > > > > > > (the >>>> > > > > > > > > >> > > > compacted >>>> > > > > > > > > >> > > > > > >> > > version). >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > On Monday, January >>>> 5, 2026 >>>> > > at >>>> > > > > > > 01:22:42 >>>> > > > > > > > > PM >>>> > > > > > > > > >> > PST, >>>> > > > > > > > > >> > > > > Lijun >>>> > > > > > > > > >> > > > > > >> Tong >>>> > > > > > > > > >> > > > > > >> > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> [email protected]> >>>> > > > wrote: >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Hello Kafka >>>> Community, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > I would like to >>>> start a >>>> > > > > discussion >>>> > > > > > > on >>>> > > > > > > > > >> > KIP-1266, >>>> > > > > > > > > >> > > > > which >>>> > > > > > > > > >> > > > > > >> > > >> proposes to >>>> > > > > > > > > >> > > > > > >> > > >> > > add >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > another new compacted >>>> > remote >>>> > > > log >>>> > > > > > > > > metadata >>>> > > > > > > > > >> > topic >>>> > > > > > > > > >> > > > for >>>> > > > > > > > > >> > > > > > the >>>> > > > > > > > > >> > > > > > >> > > tiered >>>> > > > > > > > > >> > > > > > >> > > >> > > > storage, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > limit the number of >>>> > messages >>>> > > > > that >>>> > > > > > > need >>>> > > > > > > > > to >>>> > > > > > > > > >> be >>>> > > > > > > > > >> > > > > iterated >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > > build >>>> > > > > > > > > >> > > > > > >> > > >> > the >>>> > > > > > > > > >> > > > > > >> > > >> > > > > remote >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > metadata state. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > KIP link: KIP-1266 >>>> > Bounding >>>> > > > The >>>> > > > > > > Number >>>> > > > > > > > > Of >>>> > > > > > > > > >> > > > > > >> > RemoteLogMetadata >>>> > > > > > > > > >> > > > > > >> > > >> > > Messages >>>> > > > > > > > > >> > > > > > >> > > >> > > > > via >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Compacted >>>> > RemoteLogMetadata >>>> > > > > Topic >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > < >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1266%3A+Bounding+The+Number+Of+RemoteLogMetadata+Messages+via+Compacted+Topic >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Background: >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > The current Tiered >>>> Storage >>>> > > > > > > > > implementation >>>> > > > > > > > > >> > uses >>>> > > > > > > > > >> > > a >>>> > > > > > > > > >> > > > > > >> > > >> > > > __remote_log_metadata >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > topic with infinite >>>> > > retention >>>> > > > > and >>>> > > > > > > > > >> > delete-based >>>> > > > > > > > > >> > > > > > cleanup >>>> > > > > > > > > >> > > > > > >> > > policy, >>>> > > > > > > > > >> > > > > > >> > > >> > > > causing >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > unbounded growth, >>>> slow >>>> > > broker >>>> > > > > > > > bootstrap, >>>> > > > > > > > > >> no >>>> > > > > > > > > >> > > > > mechanism >>>> > > > > > > > > >> > > > > > >> to >>>> > > > > > > > > >> > > > > > >> > > >> clean up >>>> > > > > > > > > >> > > > > > >> > > >> > > > > expired >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > segment metadata, and >>>> > > > > inefficient >>>> > > > > > > > > >> re-reading >>>> > > > > > > > > >> > > from >>>> > > > > > > > > >> > > > > > >> offset 0 >>>> > > > > > > > > >> > > > > > >> > > >> during >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > leadership changes. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Proposal: >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > A dual-topic >>>> approach that >>>> > > > > > > introduces >>>> > > > > > > > a >>>> > > > > > > > > >> new >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > __remote_log_metadata_compacted >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > topic using log >>>> compaction >>>> > > > with >>>> > > > > > > > > >> deterministic >>>> > > > > > > > > >> > > > > > >> offset-based >>>> > > > > > > > > >> > > > > > >> > > >> keys, >>>> > > > > > > > > >> > > > > > >> > > >> > > > while >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > preserving the >>>> existing >>>> > > topic >>>> > > > > for >>>> > > > > > > > audit >>>> > > > > > > > > >> > > history; >>>> > > > > > > > > >> > > > > this >>>> > > > > > > > > >> > > > > > >> > allows >>>> > > > > > > > > >> > > > > > >> > > >> > > brokers >>>> > > > > > > > > >> > > > > > >> > > >> > > > to >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > build their metadata >>>> cache >>>> > > > > > > exclusively >>>> > > > > > > > > >> from >>>> > > > > > > > > >> > the >>>> > > > > > > > > >> > > > > > >> compacted >>>> > > > > > > > > >> > > > > > >> > > >> topic, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > enables >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > cleanup of expired >>>> segment >>>> > > > > > metadata >>>> > > > > > > > > >> through >>>> > > > > > > > > >> > > > > > tombstones, >>>> > > > > > > > > >> > > > > > >> > and >>>> > > > > > > > > >> > > > > > >> > > >> > > includes >>>> > > > > > > > > >> > > > > > >> > > >> > > > a >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > migration strategy to >>>> > > populate >>>> > > > > the >>>> > > > > > > new >>>> > > > > > > > > >> topic >>>> > > > > > > > > >> > > > during >>>> > > > > > > > > >> > > > > > >> > > >> > > > upgrade—delivering >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > bounded metadata >>>> growth >>>> > and >>>> > > > > faster >>>> > > > > > > > > broker >>>> > > > > > > > > >> > > startup >>>> > > > > > > > > >> > > > > > while >>>> > > > > > > > > >> > > > > > >> > > >> > maintaining >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > backward >>>> compatibility. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > More details are in >>>> the >>>> > > > attached >>>> > > > > > KIP >>>> > > > > > > > > link. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Looking forward to >>>> your >>>> > > > > thoughts. >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Thank you for your >>>> time! >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Best, >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > Lijun Tong >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >> > > >>>> > > > > > > > > >> > > > > > >> > > >> > >>>> > > > > > > > > >> > > > > > >> > > >> >>>> > > > > > > > > >> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >> > > >>>> > > > > > > > > >> > > > > > >> > >>>> > > > > > > > > >> > > > > > >> >>>> > > > > > > > > >> > > > > > > >>>> > > > > > > > > >> > > > > > >>>> > > > > > > > > >> > > > > >>>> > > > > > > > > >> > > > >>>> > > > > > > > > >> > > >>>> > > > > > > > > >> > >>>> > > > > > > > > >> >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> >>>
