Hi Lijun, Sorry, I think more about this, and read your reply again, I think I misunderstood it.
> If the user is fine with -1 as the retention.ms, which means they're not troubled by this metadata increasing issue yet, they don't need to do the migration. I don't think this is accepted. Firstly, no one is OK with the unbounded size topic. They use -1 just because it is the default value and the disk size is still enough so far. So, everyone needs a migration plan. Therefore, the goal is: how we can make the existing retention.ms=-1 users successfully migrate to the compacted topic without data loss. Look forward to your good solution. :) Luke On Wed, Aug 19, 2026 at 11:28 AM Luke Chen <[email protected]> wrote: > Hi Lijun, > > I like the updated solution. > 1. When upgrading to the new version containing this feature, the ` > remote.log.metadata.topic.retention.ms` is still -1 (or any > existing value). The only change is that the new metadata records will be > written with a key. > 2. Then we need the detailed design to this: "we can directly do some > message format check and decide whether it's ok to make the topic > compacted". So users don't need to worry about the data loss during the > migration process. > > Look forward to the detailed design. > > Thank you, > Luke > > > On Wed, Aug 19, 2026 at 8:49 AM Lijun Tong <[email protected]> > wrote: > >> Hey Luke, >> >> Thanks for your response! >> >> > "and once the old keyless records age out, the transition to >> > compaction-based cleanup is complete." >> > >> > How will you achieve this? Could you elaborate more on this? >> >> >> I do want to point out that even without this KIP's change, currently user >> needs to set their __remote_log_metadata topic's retention time so their >> remote metadata will not increase permanently, this is recorded in the >> tiered storage KIP[1] like I excerpted below: >> >> *remote.log.metadata.topic.retention.ms >> <http://remote.log.metadata.topic.retention.ms>* >> >> Retention of the topic in milli seconds. >> >> Default: -1, that means unlimited. >> >> Users can configure this value based on their usecases. To avoid any data >> loss, this value should be more than the maximum retention period of any >> topic enabled with tiered storage in the cluster. >> >> So the old keyless records will age out naturally even we don't ask the >> user to configure retention.ms in the migration path. >> >> If the user is fine with -1 as the retention.ms, which means they're not >> troubled by this metadata increasing issue yet, they don't need to do the >> migration. >> >> If we decide to remove the retention.ms from the migration path, I don't >> envision we will need a 3 steps migration anymore, we can directly do some >> message format check and decide whether it's ok to make the topic >> compacted, the metadata remote.log.metadata.version will only have 2 >> value, >> which is 0 and 1. I think this change will answer most of the edge case >> related questions you have listed above. >> >> >> Let me know if you have other questions, thanks! >> >> [1] >> >> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/97554472/KIP-405+Kafka+Tiered+Storage#KIP405%3AKafkaTieredStorage-Configs >> >> Best, >> Lijun Tong >> >> >> Luke Chen <[email protected]> 于2026年8月18日周二 02:02写道： >> >> > Hi Lijun, >> > >> > Thanks for the reply. >> > >> > > "and once the old keyless records age out, the transition to >> > compaction-based cleanup is complete." >> > >> > How will you achieve this? Could you elaborate more on this? >> > >> > I want to make sure our migration strategy covers every edge case to >> > completely eliminate the risk of data loss, >> > >> > I think what I expect is our migration strategy covers every edge case >> to >> > completely eliminate the risk of data loss (i.e. unable to fetch from >> the >> > remote storage). So maybe you can think more about these questions >> first: >> > LC4.2: How can we avoid data loss when users directly move the version >> from >> > 0 -> 2? >> > LC4.3: How can we avoid data loss when a short retention.ms is set for >> the >> > internal topic? If it's the solution above, it needs more detailed >> > information. >> > LC4.4: How can we avoid data loss when users stay >> > in remote.log.metadata.version 1 for a long time? >> > LC4.6: How can we avoid data loss if the configs are altered >> > after remote.log.metadata.version moves to 1? >> > LC4.13. How can we avoid data loss if users are currently using >> > retention.bytes for tiered storage topics when migration? >> > >> > >> > Thank you, >> > Luke >> > >> > >> > On Wed, Aug 12, 2026 at 1:11 PM Lijun Tong <[email protected]> >> > wrote: >> > >> > > Hi Luke, >> > > >> > > Thanks for the thoughtful feedback — I appreciate you pushing on this. >> > > >> > > >> > > You raise a fair point about the distinction between user topics and >> > > internal control-plane topics like __remote_log_metadata. I agree >> that we >> > > should treat internal topics with extra care, and that relying on >> users >> > to >> > > manually coordinate retention.ms with their longest >> > remote-storage-enabled >> > > topic introduces a fragile safeguard. >> > > >> > > >> > > So here's an alternative approach: what if we simply don't ask the >> user >> > to >> > > explicitly set retention.ms on __remote_log_metadata at all during >> > > migration? Instead, we can just use whatever retention value is >> already >> > > configured on the topic. The migration would still complete naturally >> > once >> > > all the legacy keyless messages have been deleted by the existing >> > retention >> > > policy — no manual user intervention required. >> > > >> > > This way, we avoid introducing any new risk of data loss during >> > migration, >> > > since we're not asking users to change anything on the control-plane >> > topic. >> > > The migration simply works with the retention behavior that's already >> in >> > > effect, and once the old keyless records age out, the transition to >> > > compaction-based cleanup is complete. >> > > >> > > >> > > What do you think? Would this address your concern? >> > > >> > > Best, Lijun >> > > >> > > Luke Chen <[email protected]> 于2026年8月11日周二 20:25写道： >> > > >> > > > Hi Lijun, >> > > > >> > > > Thanks for the response. >> > > > >> > > > > The scenario you outlined — where a user sets retention.ms to 14 >> > days >> > > > on a >> > > > topic that has remote-storage-enabled data spanning 30 days — is >> > > actually a >> > > > risk that exists *today*, independent of this KIP. As soon as a user >> > > > enables tiered storage for a topic, they already bear the >> > responsibility >> > > of >> > > > configuring retention.ms correctly to avoid unintended data >> deletion. >> > > This >> > > > KIP doesn't introduce that risk; it inherits it from the original >> > tiered >> > > > storage design. >> > > > >> > > > I know what you mean. I think there is a key difference between >> setting >> > > > retention.ms on a user topic vs. the internal __remote_log_metadata >> > > topic: >> > > > the __remote_log_metadata is shared and cluster-wide control plane >> > topic >> > > > (that's why we call it internal topic), which serves for all tiered >> > > storage >> > > > topics. Its records are system pointers/indexes/metadata, not >> business >> > > > data. I think you know the risk, so instead of asking users to set a >> > > > retention time they like as a normal user topic, you ask users to >> set >> > > > retention.ms higher than the retention period of the longest >> > > > remote-storage-enabled topic. But then, we lose any other safeguard >> > when >> > > > users extend a user topic retention time from 14 days to 30 days, >> which >> > > > causes data loss (unable to fetch from the remote storage). This is >> > why I >> > > > think we should improve it. >> > > > >> > > > Again, we're handling the control-plane topic, which should treat it >> > more >> > > > carefully than user topics. >> > > > >> > > > > That said, I do want to highlight what this KIP is trying to >> > > accomplish: >> > > > once the migration is complete, this risk goes away. The new design >> > > > eliminates the need for operators to carefully coordinate >> retention.ms >> > > > with >> > > > their remote storage retention, because compaction can apply to new >> > data >> > > > going forward. So in a sense, the migration path is the bridge that >> > gets >> > > > users *out of* this risky situation, not into it. >> > > > >> > > > I know and thanks for the proposal for the migration path. I can >> see we >> > > > want to help users on the migration path, but we still need to >> > eliminate >> > > > the risks of data loss during migration. >> > > > >> > > > >> > > > Thank you, >> > > > Luke >> > > > >> > > > On Wed, Aug 12, 2026 at 12:15 AM Lijun Tong < >> [email protected]> >> > > > wrote: >> > > > >> > > > > Hey Luke, >> > > > > >> > > > > Thanks for the feedback — I really appreciate you taking the time >> to >> > > walk >> > > > > through that scenario. Data durability is absolutely a top >> priority, >> > > and >> > > > > I'm glad we're digging into this. >> > > > > >> > > > > I'd like to offer a slightly different perspective on the risk you >> > > > > described, though. >> > > > > >> > > > > The scenario you outlined — where a user sets retention.ms to 14 >> > days >> > > > on a >> > > > > topic that has remote-storage-enabled data spanning 30 days — is >> > > > actually a >> > > > > risk that exists *today*, independent of this KIP. As soon as a >> user >> > > > > enables tiered storage for a topic, they already bear the >> > > responsibility >> > > > of >> > > > > configuring retention.ms correctly to avoid unintended data >> > deletion. >> > > > This >> > > > > KIP doesn't introduce that risk; it inherits it from the original >> > > tiered >> > > > > storage design. >> > > > > >> > > > > If we believe this risk is too significant to ask operators to >> manage >> > > > > during a migration window, then I think we'd also need to >> acknowledge >> > > > that >> > > > > it's too significant to let operators manage in normal production >> > usage >> > > > > right now — because the exposure is the same. In both cases, a >> > > > > misconfigured retention.ms on a tiered-storage-enabled topic can >> > lead >> > > to >> > > > > data loss. >> > > > > >> > > > > That said, I do want to highlight what this KIP is trying to >> > > accomplish: >> > > > > once the migration is complete, this risk goes away. The new >> design >> > > > > eliminates the need for operators to carefully coordinate >> > retention.ms >> > > > > with >> > > > > their remote storage retention, because compaction can apply to >> new >> > > data >> > > > > going forward. So in a sense, the migration path is the bridge >> that >> > > gets >> > > > > users *out of* this risky situation, not into it. >> > > > > >> > > > > Even in the scenario where we simply respect the retention.ms >> that >> > the >> > > > > user >> > > > > has already configured — without requiring any additional manual >> > > checks — >> > > > > the end state is still strictly better than what users have today, >> > > > because >> > > > > new messages will benefit from compaction. >> > > > > >> > > > > I completely agree that we should make the migration as safe and >> > smooth >> > > > as >> > > > > possible, and I'll take a closer look at KIP-866 and KIP-848 for >> > > > > inspiration on how they handled their migration paths. If there >> are >> > > > > reasonable guardrails we can add (such as automatic configuration >> of >> > > the >> > > > > retention.ms based on current longest remote-storage-enabled >> topic's >> > > > > retention time), I'm very open to that. I just want to make sure >> > we're >> > > > > framing the risk accurately — as a pre-existing concern rather >> than >> > one >> > > > > introduced by this proposal. >> > > > > >> > > > > Happy to discuss further. Thanks again for the thoughtful review! >> > > > > >> > > > > Thanks, >> > > > > >> > > > > Lijun Tong >> > > > > >> > > > > Luke Chen <[email protected]> 于2026年8月11日周二 01:01写道： >> > > > > >> > > > > > Hi Lijun, >> > > > > > >> > > > > > Thanks for your reply. >> > > > > > I’ll send a detailed reply later when available, but I wanted to >> > > > address >> > > > > a >> > > > > > core concern first. >> > > > > > >> > > > > > About the migration strategy, we can't rely on documentation to >> > > prevent >> > > > > > potential data loss. We have to protect users from operational >> > > mistakes >> > > > > > because data durability is why users trust Kafka in the first >> > place. >> > > > > > >> > > > > > In the current design of the migration path, for example, we >> want >> > > users >> > > > > to >> > > > > > set retention.ms higher than the retention period of the >> longest >> > > > > > remote-storage-enabled topic. That means the operator needs to >> > check >> > > it >> > > > > > before migrating to version 1. It is easy for a 10 topic >> cluster. >> > But >> > > > > for a >> > > > > > hundreds of topic cluster, it'll be difficult and error-prone. >> Even >> > > if >> > > > > they >> > > > > > complete the check and set to 14 days retention time, some users >> > > could >> > > > > > still change the retention period for a remote-storage-enabled >> > topic >> > > to >> > > > > 30 >> > > > > > days later. Then 14 days later, the retention deletion kicks >> off, >> > it >> > > > will >> > > > > > cause data loss for the topics, right? Also, if the >> > > > > remote-storage-enabled >> > > > > > topic is set as retention size, not retention time, what can >> they >> > do? >> > > > > > >> > > > > > This is just an example to highlight the flaw in the current >> > design. >> > > We >> > > > > > need to rethink this path so the system can protect unexpected >> > issues >> > > > > > happening rather than leaving it as a manual effort for >> operators. >> > > > > > >> > > > > > You can check other KIPs to see how they have considered to >> achieve >> > > the >> > > > > > migration. The best example is KIP-866[1] : ZK migrating to >> KRaft. >> > > The >> > > > > > other good example is KIP-848[2]. >> > > > > > >> > > > > > I really appreciate the thought and effort put into this >> proposal! >> > > > Please >> > > > > > let me know if you have any questions, and we can discuss >> further. >> > > > > > >> > > > > > >> > > > > > [1]: >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/225153708/KIP-866+ZooKeeper+to+KRaft+Migration >> > > > > > [2] >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848%3ATheNextGenerationoftheConsumerRebalanceProtocol-SupportingOnlineConsumerGroupUpgrade >> > > > > > >> > > > > > >> > > > > > >> > > > > > Thank you, >> > > > > > Luke >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Mon, Aug 10, 2026 at 12:49 PM Lijun Tong < >> > [email protected] >> > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Hey Luke, >> > > > > > > >> > > > > > > Thanks for your response, I have answered your questions >> below. >> > > > > > > >> > > > > > > LC1. In scenario 1, we have 2 records with different keys (one >> > with >> > > > > > :UPDATE >> > > > > > > suffix), how could we achieve the final result after log >> > > compaction: >> > > > > > > Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 >> > > compacted >> > > > > > away) >> > > > > > > >> > > > > > > Current log compaction only compares the record key and keeps >> the >> > > > > latest >> > > > > > > value for the same key. So in this case, shouldn't we keep >> these >> > 2 >> > > > > > records >> > > > > > > after log compaction? >> > > > > > > >> > > > > > > *Re LC1:* This is correct. My apologies for the inconsistency >> in >> > > the >> > > > > > > proposal. I have updated the proposal to fix this issue. >> > > > > > > >> > > > > > > LC2. What does "Audit topic" mean here? >> > > > > > > >> > > > > > > *Re LC2:* Please disregard the reference to the audit topic. I >> > > > thought >> > > > > I >> > > > > > > had already refactored the proposal to remove it. >> > > > > > > >> > > > > > > For some background, my original intention was to retain the >> > > keyless >> > > > > > topic >> > > > > > > as an audit topic. However, after discussing this with others >> > > earlier >> > > > > in >> > > > > > > this email thread, we concluded that an audit topic is >> > unnecessary, >> > > > so >> > > > > I >> > > > > > > removed this concept from the design. >> > > > > > > >> > > > > > > LC3. If we already store the endOffset and brokerLeaderEpoch >> in >> > the >> > > > > > record >> > > > > > > key, why should we still need them stored in the record value? >> > > > > > > >> > > > > > > *Re LC3:* This is a good question. The reason is related to >> how >> > > > > > > RemoteLogManager currently interacts with >> RemoteMetadataManager. >> > > > > > > >> > > > > > > With the current implementation, RemoteLogManager only has >> access >> > > to >> > > > > the >> > > > > > > metadata value; it does not have the record key available at >> that >> > > > > point. >> > > > > > > TopicBasedRemoteLogMetadataManager is only one implementation >> > for >> > > > the >> > > > > > > RemoteLogMetadataManager, the key needs to be constructed from >> > the >> > > > > > metadata >> > > > > > > value rather than the other way around. >> > > > > > > >> > > > > > > For example, see: >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 >> > > > > > > >> > > > > > > LC4. About the migration: >> > > > > > > >> > > > > > > LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the >> > > difference >> > > > > > with: >> > > > > > > bin/kafka-features.sh --bootstrap-server localhost:9092 >> upgrade >> > > > > --feature >> > > > > > > remote.log.storage.version=1 >> > > > > > > >> > > > > > > *Re LC4.1:* I actually changed the feature name from >> > > > > > > remote.log.storage.version to remote.log.metadata.version, >> since >> > I >> > > > > think >> > > > > > > the latter better reflects what is being versioned. >> > > > > > > >> > > > > > > If the user runs: >> > > > > > > >> > > > > > > kafka-remote-log-metadata-migration.sh --bootstrap-server >> > > > > localhost:9092 >> > > > > > > --upgrade-to-v1 >> > > > > > > >> > > > > > > the metadata version will be upgraded from 0 to 1. >> > > > > > > >> > > > > > > If the user runs the script with --upgrade-to-v2, the metadata >> > > > version >> > > > > > will >> > > > > > > be upgraded from 1 to 2. >> > > > > > > >> > > > > > > The migration script is different from directly changing the >> > > feature >> > > > > > > version with kafka-features.sh because the migration also >> > performs >> > > > the >> > > > > > > necessary data migration and validation steps. >> > > > > > > >> > > > > > > LC4.2. If the user runs the kafka-features.sh command >> directly to >> > > > > upgrade >> > > > > > > from 0 → 2, what will happen? >> > > > > > > >> > > > > > > *Re LC4.2:* Currently it will complain about the version jump, >> > but >> > > it >> > > > > > will >> > > > > > > still proceed with the change, this means we lost all the >> keyless >> > > > > > messages >> > > > > > > and data loss. >> > > > > > > >> > > > > > > I’m considering adding a safeguard to prevent users from >> > upgrading >> > > > > > directly >> > > > > > > from version 0 to version 2 through kafka-features.sh, since >> some >> > > > > > migration >> > > > > > > steps are required between these versions should be performed >> by >> > > the >> > > > > > > migration script. Furthermore, I have documented the migration >> > > > process >> > > > > in >> > > > > > > the feature doc >> > > > > > > < >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md >> > > > > > > > >> > > > > > > and clearly directed users to use the migration tool for this >> > > > upgrade. >> > > > > > > >> > > > > > > I’m open to suggestions on how we can best ensure that users >> > follow >> > > > the >> > > > > > > migration path. >> > > > > > > >> > > > > > > LC4.3. What will happen if the setting of retention.ms is too >> > > short >> > > > > > before >> > > > > > > the existing consumers complete the fetch? >> > > > > > > >> > > > > > > *Re LC4.3:* With the current design, it is already the cluster >> > > > > operator's >> > > > > > > responsibility to configure retention.ms appropriately. If >> > > > > retention.ms >> > > > > > is >> > > > > > > too short and the metadata for an existing remote log segment >> is >> > > > > deleted >> > > > > > > before it is consumed, that metadata is lost permanently, >> which >> > can >> > > > > > result >> > > > > > > in data loss. This requirement exists independently of this >> > > migration >> > > > > and >> > > > > > > the proposed feature. And this biggest motivation for this >> > proposal >> > > > is >> > > > > to >> > > > > > > lift this burden from the kafka operator and after the >> migration, >> > > > there >> > > > > > is >> > > > > > > no such risk. >> > > > > > > >> > > > > > > The migration does not introduce a new retention requirement. >> As >> > > long >> > > > > as >> > > > > > > the operator follows the existing requirement and sets >> > > retention.ms >> > > > > > > appropriately, the migration is safe. >> > > > > > > >> > > > > > > Once the migration is complete, the keyed metadata log no >> longer >> > > > needs >> > > > > to >> > > > > > > be protected by this additional retention consideration. >> > > > > > > >> > > > > > > LC4.4. If the user sets 14 days retention.ms and upgrades to >> v1, >> > > and >> > > > > > some >> > > > > > > more metadata is added. If one month later they are still on >> v1, >> > > will >> > > > > the >> > > > > > > keyed metadata log be deleted unexpectedly due to log >> retention? >> > > > > > > >> > > > > > > *Re LC4.4:* The keyed metadata log can be truncated according >> to >> > > the >> > > > > > > configured retention policy while the cluster remains on v1. I >> > > don't >> > > > > > think >> > > > > > > this should be considered unexpected, since it is the expected >> > > > behavior >> > > > > > of >> > > > > > > the retention policy. >> > > > > > > >> > > > > > > I also don't think this creates a correctness issue, provided >> > that >> > > > > > > retention.ms should be configured to be at least as long as >> the >> > > > > > retention >> > > > > > > period of the longest remote-storage-enabled topic. Under that >> > > > > condition, >> > > > > > > even if some older keyed metadata records are removed, they >> will >> > no >> > > > > > longer >> > > > > > > be needed by the remote storage system. >> > > > > > > >> > > > > > > >> > > > > > > LC4.6. "Log cleaner won't compact yet (blocked by >> > > > > min.compaction.lag.ms >> > > > > > )", >> > > > > > > what will happen if a user or operator changes the config to a >> > > > shorter >> > > > > > > value? >> > > > > > > >> > > > > > > *Re LC4.6:* The migration script will set >> min.compaction.lag.ms >> > to >> > > > the >> > > > > > > same >> > > > > > > value as retention.ms. Therefore, if the operator changes the >> > > > > > > configuration >> > > > > > > before running the migration, the migration script will >> overwrite >> > > > that >> > > > > > > value. >> > > > > > > >> > > > > > > However, if the operator changes min.compaction.lag.ms after >> the >> > > > > > migration >> > > > > > > has started and sets it to a value that is shorter than the >> > > required >> > > > > > > period, metadata associated with active remote log segments >> could >> > > be >> > > > > > > compacted away prematurely. I will clarify this requirement in >> > the >> > > > KIP. >> > > > > > > >> > > > > > > LC4.7. In "upgrade-to-v2", what does "Tool displays retention >> > > > reminder >> > > > > > and >> > > > > > > checks if enough time has passed" mean? >> > > > > > > >> > > > > > > *Re LC4.7:* When migrating from remote.log.metadata.version=1 >> to >> > 2, >> > > > the >> > > > > > > migration script will scan the topic and verify whether all >> > > existing >> > > > > > > messages have keys. >> > > > > > > >> > > > > > > If it finds keyless messages, it will record the latest >> timestamp >> > > > among >> > > > > > > those messages. Based on that timestamp and the topic's >> > > retention.ms >> > > > , >> > > > > > the >> > > > > > > tool can calculate an estimated time when the last keyless >> > message >> > > > > should >> > > > > > > have been removed by retention. >> > > > > > > >> > > > > > > The tool will display this information to the user. The user >> can >> > > then >> > > > > > rerun >> > > > > > > the migration after that estimated time and proceed to v2 once >> > the >> > > > > > keyless >> > > > > > > messages have expired. >> > > > > > > >> > > > > > > LC4.8. In "upgrade-to-v2", "Scans topic for null-key >> messages", >> > > does >> > > > > that >> > > > > > > mean we'll launch a consumer and read the whole log of the >> > > > > > > __remote_log_metadata topic? If each broker already loads all >> > logs >> > > in >> > > > > > some >> > > > > > > __remote_log_metadata partitions, can't we do the scan during >> > > loading >> > > > > and >> > > > > > > directly return the result to the admin client? >> > > > > > > >> > > > > > > *Re LC4.8:* This is true: each broker already loads all >> records >> > > from >> > > > > its >> > > > > > > assigned partitions of __remote_log_metadata during startup. >> > > > > > > >> > > > > > > However, the validation required for the migration—checking >> > whether >> > > > > > records >> > > > > > > are keyed and, for keyless records, determining their >> > timestamps—is >> > > > > > > unrelated to the existing loading logic. >> > > > > > > >> > > > > > > I would prefer not to introduce migration-specific logic into >> the >> > > > > > existing >> > > > > > > metadata loading path and then remove or maintain that logic >> > after >> > > > the >> > > > > > > migration is complete. Keeping the migration scan separate >> makes >> > > the >> > > > > > > existing runtime path simpler and avoids introducing >> additional >> > > > > > complexity >> > > > > > > into the first version of this feature. >> > > > > > > >> > > > > > > LC4.9. Could you show the command result of >> > > > > > > kafka-remote-log-metadata-migration.sh in the KIP? >> > > > > > > >> > > > > > > *Re LC4.9:* Updated. Thanks for the suggestion! >> > > > > > > >> > > > > > > LC4.10. Also, could you show the --help result of >> > > > > > > kafka-remote-log-metadata-migration.sh? >> > > > > > > >> > > > > > > *Re LC4.10:* Updated. Thanks for the suggestion! >> > > > > > > >> > > > > > > LC4.11. Have you considered how a Kubernetes operator can >> > automate >> > > > the >> > > > > > > migration steps? >> > > > > > > >> > > > > > > *Re LC4.11:* I’ll need some time to investigate this and >> > understand >> > > > the >> > > > > > > implications for Kubernetes operators. I’ll follow up once >> I’ve >> > > had a >> > > > > > > chance to look into it in more detail. >> > > > > > > >> > > > > > > LC4.12. I think the migration plan is the most critical part >> of >> > > this >> > > > > KIP. >> > > > > > > Users cannot accept any metadata deleted before it is handled >> > > > > completely. >> > > > > > > We have to make sure all the edge cases are covered before >> moving >> > > > this >> > > > > > > forward. >> > > > > > > >> > > > > > > *Re LC4.12:* I agree that the migration plan is one of the >> most >> > > > > critical >> > > > > > > parts of this KIP. >> > > > > > > >> > > > > > > With the current migration design, the key requirement is that >> > > > > > > retention.ms >> > > > > > > must not be shorter than the retention period of the longest >> > > > > > > remote-storage-enabled topic. As long as this requirement is >> > > > satisfied, >> > > > > > the >> > > > > > > migration does not introduce any additional data-loss scenario >> > > beyond >> > > > > the >> > > > > > > requirements of the existing remote storage implementation. >> > > > > > > >> > > > > > > I will make this requirement more explicit in the migration >> > section >> > > > and >> > > > > > > clarify the relevant edge cases. >> > > > > > > >> > > > > > > Let me know if you have more questions, thanks! >> > > > > > > >> > > > > > > >> > > > > > > Best, >> > > > > > > >> > > > > > > Lijun Tong >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
