Hi Josep, This would be a fascinating feature, some well known Kafka users are using Kafka in a cloud-native env. As for as I know, there are already some secondary development version Kafka which provide this feature, for example, I am using AutoMq(https://github.com/AutoMQ/automq) in my environment, which significantly helped ms reduced the cost, so I think it’s worthwhile to clarify some related details: 1. Is this feature available by just a minor adjust of config or it will intrude current code heavily, say, AutoMq is 100% compatible with Kafka and doesn’t intrude the code heavily 2. Though we are not discussing implement details, it’s worth giving some high-level architecture ideas, and it’s better to compare with AutoMq like systems. 3. What we will provide through it, I think we will just provide a common interface and put implementations in another repos, just as we did for Kafka Connect and Kafka Tired Storage. 4. How to deal with KRaft related protocol, since metadata topic is managed differently with __cluster_metadata, through this KIP, will we align the gap between __cluster_metadata and data topics by put metadata in an object storage? if so, there will be no standby controller? since standby controller is the __cluster_metadata followers and there will be no followers.
— Ziming > On Apr 16, 2025, at 19:58, Josep Prat <josep.p...@aiven.io.INVALID> wrote: > > Hi Kafka Devs! > > We want to start a new KIP discussion about introducing a new type of > topics that would make use of Object Storage as the primary source of > storage. However, as this KIP is big we decided to split it into multiple > related KIPs. > We have the motivational KIP-1150 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics) > that aims to discuss if Apache Kafka should aim to have this type of > feature at all. This KIP doesn't go onto details on how to implement it. > This follows the same approach used when we discussed KRaft. > > But as we know that it is sometimes really hard to discuss on that meta > level, we also created several sub-kips (linked in KIP-1150) that offer an > implementation of this feature. > > We kindly ask you to use the proper DISCUSS threads for each type of > concern and keep this one to discuss whether Apache Kafka wants to have > this feature or not. > > Thanks in advance on behalf of all the authors of this KIP. > > ------------------ > Josep Prat > Open Source Engineering Director, Aiven > josep.p...@aiven.io | +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > Anna Richardson, Kenneth Chen > Amtsgericht Charlottenburg, HRB 209739 B
