On Tue, May 13, 2025, at 19:34, Jun Rao wrote: > JR3. "Permit multi-region active-active topics with automatic failover". > Could you elaborate on the benefit of this? Cloud providers still charge > cross region data transfer in object stores, right?
First, we should have emphasized in the KIP that this is targeted to multi-region stretch clusters. You are right, all the associated cloud costs apply to cross-region replication. However, this is still beneficial. The main benefit is that it’s a new (or improved) capability. Cross-regional replication of diskless topics is similar to MirrorMaker, but better in some aspects: 1. It's potentially faster and easier operationally, because the data are replicated using internal systems of the cloud provider without the Kafka producer-consumer protocol. 2. For the same reasons, metadata like record offsets could be precisely preserved. It can also be compared to stretch clusters with classic topics, where the main benefit is cost reduction: replication between regions happens only once and doesn’t depend on the replication factor and replica placement. Best, Ivan > > On Wed, Apr 16, 2025 at 5:00 AM Josep Prat <josep.p...@aiven.io.invalid> > wrote: > > > Hi Kafka Devs! > > > > We want to start a new KIP discussion about introducing a new type of > > topics that would make use of Object Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > ) > > that aims to discuss if Apache Kafka should aim to have this type of > > feature at all. This KIP doesn't go onto details on how to implement it. > > This follows the same approach used when we discussed KRaft. > > > > But as we know that it is sometimes really hard to discuss on that meta > > level, we also created several sub-kips (linked in KIP-1150) that offer an > > implementation of this feature. > > > > We kindly ask you to use the proper DISCUSS threads for each type of > > concern and keep this one to discuss whether Apache Kafka wants to have > > this feature or not. > > > > Thanks in advance on behalf of all the authors of this KIP. > > > > ------------------ > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
