Hey Luke, Thanks for your response!
> "and once the old keyless records age out, the transition to > compaction-based cleanup is complete." > > How will you achieve this? Could you elaborate more on this? I do want to point out that even without this KIP's change, currently user needs to set their __remote_log_metadata topic's retention time so their remote metadata will not increase permanently, this is recorded in the tiered storage KIP[1] like I excerpted below: *remote.log.metadata.topic.retention.ms <http://remote.log.metadata.topic.retention.ms>* Retention of the topic in milli seconds. Default: -1, that means unlimited. Users can configure this value based on their usecases. To avoid any data loss, this value should be more than the maximum retention period of any topic enabled with tiered storage in the cluster. So the old keyless records will age out naturally even we don't ask the user to configure retention.ms in the migration path. If the user is fine with -1 as the retention.ms, which means they're not troubled by this metadata increasing issue yet, they don't need to do the migration. If we decide to remove the retention.ms from the migration path, I don't envision we will need a 3 steps migration anymore, we can directly do some message format check and decide whether it's ok to make the topic compacted, the metadata remote.log.metadata.version will only have 2 value, which is 0 and 1. I think this change will answer most of the edge case related questions you have listed above. Let me know if you have other questions, thanks! [1] https://cwiki.apache.org/confluence/spaces/KAFKA/pages/97554472/KIP-405+Kafka+Tiered+Storage#KIP405%3AKafkaTieredStorage-Configs Best, Lijun Tong Luke Chen <[email protected]> 于2026年8月18日周二 02:02写道： > Hi Lijun, > > Thanks for the reply. > > > "and once the old keyless records age out, the transition to > compaction-based cleanup is complete." > > How will you achieve this? Could you elaborate more on this? > > I want to make sure our migration strategy covers every edge case to > completely eliminate the risk of data loss, > > I think what I expect is our migration strategy covers every edge case to > completely eliminate the risk of data loss (i.e. unable to fetch from the > remote storage). So maybe you can think more about these questions first: > LC4.2: How can we avoid data loss when users directly move the version from > 0 -> 2? > LC4.3: How can we avoid data loss when a short retention.ms is set for the > internal topic? If it's the solution above, it needs more detailed > information. > LC4.4: How can we avoid data loss when users stay > in remote.log.metadata.version 1 for a long time? > LC4.6: How can we avoid data loss if the configs are altered > after remote.log.metadata.version moves to 1? > LC4.13. How can we avoid data loss if users are currently using > retention.bytes for tiered storage topics when migration? > > > Thank you, > Luke > > > On Wed, Aug 12, 2026 at 1:11 PM Lijun Tong <[email protected]> > wrote: > > > Hi Luke, > > > > Thanks for the thoughtful feedback — I appreciate you pushing on this. > > > > > > You raise a fair point about the distinction between user topics and > > internal control-plane topics like __remote_log_metadata. I agree that we > > should treat internal topics with extra care, and that relying on users > to > > manually coordinate retention.ms with their longest > remote-storage-enabled > > topic introduces a fragile safeguard. > > > > > > So here's an alternative approach: what if we simply don't ask the user > to > > explicitly set retention.ms on __remote_log_metadata at all during > > migration? Instead, we can just use whatever retention value is already > > configured on the topic. The migration would still complete naturally > once > > all the legacy keyless messages have been deleted by the existing > retention > > policy — no manual user intervention required. > > > > This way, we avoid introducing any new risk of data loss during > migration, > > since we're not asking users to change anything on the control-plane > topic. > > The migration simply works with the retention behavior that's already in > > effect, and once the old keyless records age out, the transition to > > compaction-based cleanup is complete. > > > > > > What do you think? Would this address your concern? > > > > Best, Lijun > > > > Luke Chen <[email protected]> 于2026年8月11日周二 20:25写道： > > > > > Hi Lijun, > > > > > > Thanks for the response. > > > > > > > The scenario you outlined — where a user sets retention.ms to 14 > days > > > on a > > > topic that has remote-storage-enabled data spanning 30 days — is > > actually a > > > risk that exists *today*, independent of this KIP. As soon as a user > > > enables tiered storage for a topic, they already bear the > responsibility > > of > > > configuring retention.ms correctly to avoid unintended data deletion. > > This > > > KIP doesn't introduce that risk; it inherits it from the original > tiered > > > storage design. > > > > > > I know what you mean. I think there is a key difference between setting > > > retention.ms on a user topic vs. the internal __remote_log_metadata > > topic: > > > the __remote_log_metadata is shared and cluster-wide control plane > topic > > > (that's why we call it internal topic), which serves for all tiered > > storage > > > topics. Its records are system pointers/indexes/metadata, not business > > > data. I think you know the risk, so instead of asking users to set a > > > retention time they like as a normal user topic, you ask users to set > > > retention.ms higher than the retention period of the longest > > > remote-storage-enabled topic. But then, we lose any other safeguard > when > > > users extend a user topic retention time from 14 days to 30 days, which > > > causes data loss (unable to fetch from the remote storage). This is > why I > > > think we should improve it. > > > > > > Again, we're handling the control-plane topic, which should treat it > more > > > carefully than user topics. > > > > > > > That said, I do want to highlight what this KIP is trying to > > accomplish: > > > once the migration is complete, this risk goes away. The new design > > > eliminates the need for operators to carefully coordinate retention.ms > > > with > > > their remote storage retention, because compaction can apply to new > data > > > going forward. So in a sense, the migration path is the bridge that > gets > > > users *out of* this risky situation, not into it. > > > > > > I know and thanks for the proposal for the migration path. I can see we > > > want to help users on the migration path, but we still need to > eliminate > > > the risks of data loss during migration. > > > > > > > > > Thank you, > > > Luke > > > > > > On Wed, Aug 12, 2026 at 12:15 AM Lijun Tong <[email protected]> > > > wrote: > > > > > > > Hey Luke, > > > > > > > > Thanks for the feedback — I really appreciate you taking the time to > > walk > > > > through that scenario. Data durability is absolutely a top priority, > > and > > > > I'm glad we're digging into this. > > > > > > > > I'd like to offer a slightly different perspective on the risk you > > > > described, though. > > > > > > > > The scenario you outlined — where a user sets retention.ms to 14 > days > > > on a > > > > topic that has remote-storage-enabled data spanning 30 days — is > > > actually a > > > > risk that exists *today*, independent of this KIP. As soon as a user > > > > enables tiered storage for a topic, they already bear the > > responsibility > > > of > > > > configuring retention.ms correctly to avoid unintended data > deletion. > > > This > > > > KIP doesn't introduce that risk; it inherits it from the original > > tiered > > > > storage design. > > > > > > > > If we believe this risk is too significant to ask operators to manage > > > > during a migration window, then I think we'd also need to acknowledge > > > that > > > > it's too significant to let operators manage in normal production > usage > > > > right now — because the exposure is the same. In both cases, a > > > > misconfigured retention.ms on a tiered-storage-enabled topic can > lead > > to > > > > data loss. > > > > > > > > That said, I do want to highlight what this KIP is trying to > > accomplish: > > > > once the migration is complete, this risk goes away. The new design > > > > eliminates the need for operators to carefully coordinate > retention.ms > > > > with > > > > their remote storage retention, because compaction can apply to new > > data > > > > going forward. So in a sense, the migration path is the bridge that > > gets > > > > users *out of* this risky situation, not into it. > > > > > > > > Even in the scenario where we simply respect the retention.ms that > the > > > > user > > > > has already configured — without requiring any additional manual > > checks — > > > > the end state is still strictly better than what users have today, > > > because > > > > new messages will benefit from compaction. > > > > > > > > I completely agree that we should make the migration as safe and > smooth > > > as > > > > possible, and I'll take a closer look at KIP-866 and KIP-848 for > > > > inspiration on how they handled their migration paths. If there are > > > > reasonable guardrails we can add (such as automatic configuration of > > the > > > > retention.ms based on current longest remote-storage-enabled topic's > > > > retention time), I'm very open to that. I just want to make sure > we're > > > > framing the risk accurately — as a pre-existing concern rather than > one > > > > introduced by this proposal. > > > > > > > > Happy to discuss further. Thanks again for the thoughtful review! > > > > > > > > Thanks, > > > > > > > > Lijun Tong > > > > > > > > Luke Chen <[email protected]> 于2026年8月11日周二 01:01写道： > > > > > > > > > Hi Lijun, > > > > > > > > > > Thanks for your reply. > > > > > I’ll send a detailed reply later when available, but I wanted to > > > address > > > > a > > > > > core concern first. > > > > > > > > > > About the migration strategy, we can't rely on documentation to > > prevent > > > > > potential data loss. We have to protect users from operational > > mistakes > > > > > because data durability is why users trust Kafka in the first > place. > > > > > > > > > > In the current design of the migration path, for example, we want > > users > > > > to > > > > > set retention.ms higher than the retention period of the longest > > > > > remote-storage-enabled topic. That means the operator needs to > check > > it > > > > > before migrating to version 1. It is easy for a 10 topic cluster. > But > > > > for a > > > > > hundreds of topic cluster, it'll be difficult and error-prone. Even > > if > > > > they > > > > > complete the check and set to 14 days retention time, some users > > could > > > > > still change the retention period for a remote-storage-enabled > topic > > to > > > > 30 > > > > > days later. Then 14 days later, the retention deletion kicks off, > it > > > will > > > > > cause data loss for the topics, right? Also, if the > > > > remote-storage-enabled > > > > > topic is set as retention size, not retention time, what can they > do? > > > > > > > > > > This is just an example to highlight the flaw in the current > design. > > We > > > > > need to rethink this path so the system can protect unexpected > issues > > > > > happening rather than leaving it as a manual effort for operators. > > > > > > > > > > You can check other KIPs to see how they have considered to achieve > > the > > > > > migration. The best example is KIP-866[1] : ZK migrating to KRaft. > > The > > > > > other good example is KIP-848[2]. > > > > > > > > > > I really appreciate the thought and effort put into this proposal! > > > Please > > > > > let me know if you have any questions, and we can discuss further. > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/225153708/KIP-866+ZooKeeper+to+KRaft+Migration > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848%3ATheNextGenerationoftheConsumerRebalanceProtocol-SupportingOnlineConsumerGroupUpgrade > > > > > > > > > > > > > > > > > > > > Thank you, > > > > > Luke > > > > > > > > > > > > > > > > > > > > On Mon, Aug 10, 2026 at 12:49 PM Lijun Tong < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Hey Luke, > > > > > > > > > > > > Thanks for your response, I have answered your questions below. > > > > > > > > > > > > LC1. In scenario 1, we have 2 records with different keys (one > with > > > > > :UPDATE > > > > > > suffix), how could we achieve the final result after log > > compaction: > > > > > > Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 > > compacted > > > > > away) > > > > > > > > > > > > Current log compaction only compares the record key and keeps the > > > > latest > > > > > > value for the same key. So in this case, shouldn't we keep these > 2 > > > > > records > > > > > > after log compaction? > > > > > > > > > > > > *Re LC1:* This is correct. My apologies for the inconsistency in > > the > > > > > > proposal. I have updated the proposal to fix this issue. > > > > > > > > > > > > LC2. What does "Audit topic" mean here? > > > > > > > > > > > > *Re LC2:* Please disregard the reference to the audit topic. I > > > thought > > > > I > > > > > > had already refactored the proposal to remove it. > > > > > > > > > > > > For some background, my original intention was to retain the > > keyless > > > > > topic > > > > > > as an audit topic. However, after discussing this with others > > earlier > > > > in > > > > > > this email thread, we concluded that an audit topic is > unnecessary, > > > so > > > > I > > > > > > removed this concept from the design. > > > > > > > > > > > > LC3. If we already store the endOffset and brokerLeaderEpoch in > the > > > > > record > > > > > > key, why should we still need them stored in the record value? > > > > > > > > > > > > *Re LC3:* This is a good question. The reason is related to how > > > > > > RemoteLogManager currently interacts with RemoteMetadataManager. > > > > > > > > > > > > With the current implementation, RemoteLogManager only has access > > to > > > > the > > > > > > metadata value; it does not have the record key available at that > > > > point. > > > > > > TopicBasedRemoteLogMetadataManager is only one implementation > for > > > the > > > > > > RemoteLogMetadataManager, the key needs to be constructed from > the > > > > > metadata > > > > > > value rather than the other way around. > > > > > > > > > > > > For example, see: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 > > > > > > > > > > > > LC4. About the migration: > > > > > > > > > > > > LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the > > difference > > > > > with: > > > > > > bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade > > > > --feature > > > > > > remote.log.storage.version=1 > > > > > > > > > > > > *Re LC4.1:* I actually changed the feature name from > > > > > > remote.log.storage.version to remote.log.metadata.version, since > I > > > > think > > > > > > the latter better reflects what is being versioned. > > > > > > > > > > > > If the user runs: > > > > > > > > > > > > kafka-remote-log-metadata-migration.sh --bootstrap-server > > > > localhost:9092 > > > > > > --upgrade-to-v1 > > > > > > > > > > > > the metadata version will be upgraded from 0 to 1. > > > > > > > > > > > > If the user runs the script with --upgrade-to-v2, the metadata > > > version > > > > > will > > > > > > be upgraded from 1 to 2. > > > > > > > > > > > > The migration script is different from directly changing the > > feature > > > > > > version with kafka-features.sh because the migration also > performs > > > the > > > > > > necessary data migration and validation steps. > > > > > > > > > > > > LC4.2. If the user runs the kafka-features.sh command directly to > > > > upgrade > > > > > > from 0 → 2, what will happen? > > > > > > > > > > > > *Re LC4.2:* Currently it will complain about the version jump, > but > > it > > > > > will > > > > > > still proceed with the change, this means we lost all the keyless > > > > > messages > > > > > > and data loss. > > > > > > > > > > > > I’m considering adding a safeguard to prevent users from > upgrading > > > > > directly > > > > > > from version 0 to version 2 through kafka-features.sh, since some > > > > > migration > > > > > > steps are required between these versions should be performed by > > the > > > > > > migration script. Furthermore, I have documented the migration > > > process > > > > in > > > > > > the feature doc > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md > > > > > > > > > > > > > and clearly directed users to use the migration tool for this > > > upgrade. > > > > > > > > > > > > I’m open to suggestions on how we can best ensure that users > follow > > > the > > > > > > migration path. > > > > > > > > > > > > LC4.3. What will happen if the setting of retention.ms is too > > short > > > > > before > > > > > > the existing consumers complete the fetch? > > > > > > > > > > > > *Re LC4.3:* With the current design, it is already the cluster > > > > operator's > > > > > > responsibility to configure retention.ms appropriately. If > > > > retention.ms > > > > > is > > > > > > too short and the metadata for an existing remote log segment is > > > > deleted > > > > > > before it is consumed, that metadata is lost permanently, which > can > > > > > result > > > > > > in data loss. This requirement exists independently of this > > migration > > > > and > > > > > > the proposed feature. And this biggest motivation for this > proposal > > > is > > > > to > > > > > > lift this burden from the kafka operator and after the migration, > > > there > > > > > is > > > > > > no such risk. > > > > > > > > > > > > The migration does not introduce a new retention requirement. As > > long > > > > as > > > > > > the operator follows the existing requirement and sets > > retention.ms > > > > > > appropriately, the migration is safe. > > > > > > > > > > > > Once the migration is complete, the keyed metadata log no longer > > > needs > > > > to > > > > > > be protected by this additional retention consideration. > > > > > > > > > > > > LC4.4. If the user sets 14 days retention.ms and upgrades to v1, > > and > > > > > some > > > > > > more metadata is added. If one month later they are still on v1, > > will > > > > the > > > > > > keyed metadata log be deleted unexpectedly due to log retention? > > > > > > > > > > > > *Re LC4.4:* The keyed metadata log can be truncated according to > > the > > > > > > configured retention policy while the cluster remains on v1. I > > don't > > > > > think > > > > > > this should be considered unexpected, since it is the expected > > > behavior > > > > > of > > > > > > the retention policy. > > > > > > > > > > > > I also don't think this creates a correctness issue, provided > that > > > > > > retention.ms should be configured to be at least as long as the > > > > > retention > > > > > > period of the longest remote-storage-enabled topic. Under that > > > > condition, > > > > > > even if some older keyed metadata records are removed, they will > no > > > > > longer > > > > > > be needed by the remote storage system. > > > > > > > > > > > > > > > > > > LC4.6. "Log cleaner won't compact yet (blocked by > > > > min.compaction.lag.ms > > > > > )", > > > > > > what will happen if a user or operator changes the config to a > > > shorter > > > > > > value? > > > > > > > > > > > > *Re LC4.6:* The migration script will set min.compaction.lag.ms > to > > > the > > > > > > same > > > > > > value as retention.ms. Therefore, if the operator changes the > > > > > > configuration > > > > > > before running the migration, the migration script will overwrite > > > that > > > > > > value. > > > > > > > > > > > > However, if the operator changes min.compaction.lag.ms after the > > > > > migration > > > > > > has started and sets it to a value that is shorter than the > > required > > > > > > period, metadata associated with active remote log segments could > > be > > > > > > compacted away prematurely. I will clarify this requirement in > the > > > KIP. > > > > > > > > > > > > LC4.7. In "upgrade-to-v2", what does "Tool displays retention > > > reminder > > > > > and > > > > > > checks if enough time has passed" mean? > > > > > > > > > > > > *Re LC4.7:* When migrating from remote.log.metadata.version=1 to > 2, > > > the > > > > > > migration script will scan the topic and verify whether all > > existing > > > > > > messages have keys. > > > > > > > > > > > > If it finds keyless messages, it will record the latest timestamp > > > among > > > > > > those messages. Based on that timestamp and the topic's > > retention.ms > > > , > > > > > the > > > > > > tool can calculate an estimated time when the last keyless > message > > > > should > > > > > > have been removed by retention. > > > > > > > > > > > > The tool will display this information to the user. The user can > > then > > > > > rerun > > > > > > the migration after that estimated time and proceed to v2 once > the > > > > > keyless > > > > > > messages have expired. > > > > > > > > > > > > LC4.8. In "upgrade-to-v2", "Scans topic for null-key messages", > > does > > > > that > > > > > > mean we'll launch a consumer and read the whole log of the > > > > > > __remote_log_metadata topic? If each broker already loads all > logs > > in > > > > > some > > > > > > __remote_log_metadata partitions, can't we do the scan during > > loading > > > > and > > > > > > directly return the result to the admin client? > > > > > > > > > > > > *Re LC4.8:* This is true: each broker already loads all records > > from > > > > its > > > > > > assigned partitions of __remote_log_metadata during startup. > > > > > > > > > > > > However, the validation required for the migration—checking > whether > > > > > records > > > > > > are keyed and, for keyless records, determining their > timestamps—is > > > > > > unrelated to the existing loading logic. > > > > > > > > > > > > I would prefer not to introduce migration-specific logic into the > > > > > existing > > > > > > metadata loading path and then remove or maintain that logic > after > > > the > > > > > > migration is complete. Keeping the migration scan separate makes > > the > > > > > > existing runtime path simpler and avoids introducing additional > > > > > complexity > > > > > > into the first version of this feature. > > > > > > > > > > > > LC4.9. Could you show the command result of > > > > > > kafka-remote-log-metadata-migration.sh in the KIP? > > > > > > > > > > > > *Re LC4.9:* Updated. Thanks for the suggestion! > > > > > > > > > > > > LC4.10. Also, could you show the --help result of > > > > > > kafka-remote-log-metadata-migration.sh? > > > > > > > > > > > > *Re LC4.10:* Updated. Thanks for the suggestion! > > > > > > > > > > > > LC4.11. Have you considered how a Kubernetes operator can > automate > > > the > > > > > > migration steps? > > > > > > > > > > > > *Re LC4.11:* I’ll need some time to investigate this and > understand > > > the > > > > > > implications for Kubernetes operators. I’ll follow up once I’ve > > had a > > > > > > chance to look into it in more detail. > > > > > > > > > > > > LC4.12. I think the migration plan is the most critical part of > > this > > > > KIP. > > > > > > Users cannot accept any metadata deleted before it is handled > > > > completely. > > > > > > We have to make sure all the edge cases are covered before moving > > > this > > > > > > forward. > > > > > > > > > > > > *Re LC4.12:* I agree that the migration plan is one of the most > > > > critical > > > > > > parts of this KIP. > > > > > > > > > > > > With the current migration design, the key requirement is that > > > > > > retention.ms > > > > > > must not be shorter than the retention period of the longest > > > > > > remote-storage-enabled topic. As long as this requirement is > > > satisfied, > > > > > the > > > > > > migration does not introduce any additional data-loss scenario > > beyond > > > > the > > > > > > requirements of the existing remote storage implementation. > > > > > > > > > > > > I will make this requirement more explicit in the migration > section > > > and > > > > > > clarify the relevant edge cases. > > > > > > > > > > > > Let me know if you have more questions, thanks! > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Lijun Tong > > > > > > > > > > > > > > > > > > > > >
