For shortcoming (2), looking at the code, as long as the partition is not offline during disablement (leader is available), then the remote-log deletion might proceed as expected. Note that the feature is not tested out in ZK mode, and the implementation details can change later. So, I'd suggest you upgrade to KRaft.
On Sat, Nov 2, 2024 at 4:15 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jianfeng, > > As Luke already mentioned, the feature was not tested in ZK mode. > If you still want to enable it in the ZK mode, then: > > 1. Disable the LogConfig#validateNoInvalidRemoteStorageConfigsInZK > <https://sourcegraph.com/github.com/apache/kafka@3.9/-/blob/storage/src/main/java/org/apache/kafka/storage/internals/log/LogConfig.java?L675-682> > validation. > 2. Update the relevant unit tests, otherwise build will fail. > 3. Enable the integration test DisableRemoteLogOnTopicTest > <https://sourcegraph.com/github.com/apache/kafka@3.9/-/blob/storage/src/test/java/org/apache/kafka/tiered/storage/integration/DisableRemoteLogOnTopicTest.java?L44-48> > for > both Zk and Kraft modes. > > *Shortcomings with ZK mode* > 1. The disabling remote storage validation is done by the controller only > in the Kraft mode. Zk mode doesn't support it. > Refer to #16693 <https://github.com/apache/kafka/pull/16693> PR and > RemoteTopicCrudTest#testUpdateTopicConfigWithDisablingRemoteStorage > <https://sourcegraph.com/github.com/apache/kafka@3.9/-/blob/core/src/test/scala/integration/kafka/admin/RemoteTopicCrudTest.scala?L428-445> > test. > 2. There can be few edge-cases which may not be covered/tested: (eg) > replica down during disablement, > replica was down during disablement followed by enablement, and so on. > > Read the discussion and voting thread for more details. > > Thanks, > Kamal > > On Wed, Oct 30, 2024 at 12:56 PM Luke Chen <show...@gmail.com> wrote: > >> Hi Jianfeng, >> >> The reason we don't support ZK mode is because the ZK is going to be >> removed in v4.0.0, so we decided to only implement it in KRaft mode. >> >> > I read the source code and seems kafka broker only updates the LogConfig >> and then invokes maybeUpdateRemoteLogComponents to >> cancel some tasks. So I want to know how if just update the log configs >> without a v5 stopReplicas v5 rpc from controller to broker in zk mode? >> >> The ZK mode is not completely implemented IIRC, and not being tested. So >> I'd suggest you upgrade to KRaft for it. >> >> Thank you. >> Luke >> >> On Mon, Oct 28, 2024 at 12:32 PM Jianfeng Lin >> <jianfeng....@shopee.com.invalid> wrote: >> >> > Hi kafka community, I read the KIP-950 >> > < >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-ZookeeperBackedCluster >> > > >> > which >> > describe the disablement of kafka tiered storage. It said that zk mode >> > won't be supported and I can't find out the reason, with confusion, I >> have >> > a question: >> > This KIP proposes an implement with stopReplicas v5 to let brokers >> know >> > which topic partitions should stop their remote storage functions when >> > "remote.log.copy.disable" is set to true, different from kraft mode, it >> > takes an extra rpc round. I read the source code and seems kafka broker >> > only updates the LogConfig and then invokes >> maybeUpdateRemoteLogComponents >> > to >> > cancel some tasks. So I want to know how if just update the log configs >> > without a v5 stopReplicas v5 rpc from controller to broker in zk mode? >> And >> > why zk mode is not supported to disable? >> > Really appreciate if Luke Chen<show...@gmail.com> could help to >> answer, >> > thanks in advance! >> > >> > >> > Jianfeng Lin from Shopee >> > >> >
