Hi Luke, Thank you for creating this dedicated thread; we definitely need a space to discuss future steps for these topics. I apologize for my delay on KIP-1183 and will provide more details in the coming weeks.
I agree with Stanislav that we should first focus on the community's direction. Specifically, should we consider introducing a leaderless architecture to Kafka, given that it currently relies on a partitioned, leader-based model? >From my own perspective, I’m particularly interested in how Leaderless and Leader-based architectures differ when it comes to handling data locality—which directly affects batching and fetch efficiency—and in the way core features are implemented. For instance, ordering, compaction, transactions, idempotent producers, and queues all have to be realized on the Coordinator in a Leaderless design, whereas in a Leader-based design they are handled by the Leader Partition. If the community plans to adopt a leaderless architecture, will the focus be on a complete transition to leaderless, or will both architectures coexist in the long term? I welcome discussions on this topic and am eager to hear diverse opinions. Regards, Xinyu On Wed, Aug 6, 2025 at 3:05 AM Stanislav Kozlovski < stanislavkozlov...@apache.org> wrote: > Thank you Luke for this wonderful summary and taking initiative. > > To me, it seems like a large differentiator from KIP-1150 and others is > the leaderless design. The other two don’t allow for it. > > It sounds productive to focus the discussion on whether the leaderless > design is worth it on top of the replication cost savings. > > I’m of the opinion that it’s worth pursuing - both for the truly zero > network cost (no producer cross az) but perhaps even more importantly the > zero state architecture that promises to significantly simplify operations, > including auto scaling brokers and scaling throughput per partition > > It would be great if the folks at Aiven could address the concerns > regarding queue and transactions support. I’m not of the opinion that these > things need to ship with v1, but it would be wise to ensure nothing in the > architecture blocks these features from being shipped in the future > > KIP-1176 is also very cool, addressing the acks=1 case will still be > necessary. I think it’s a necessary feature to implement, but I’d be > disappointed if that’s the only diskless solution the community agrees on. > > A good path, if possible, may be to merge KIP-1150 and KIP-1176. > > If instead the community decides leaderless isn’t necessary, then KIP-1183 > seems fit. > > That’s my opinion. Happy to hear if anyone disagrees. > > On 2025/08/05 14:30:45 Josep Prat wrote: > > Hi Luke and community! > > > > Luke, thank you for being proactive and caring about this topic! > > > > In the meantime we have been keeping ourselves busy pushing our > > implementation of KIP-1150 to production to validate our assumptions and > > confirm its strengths while discovering its weaknesses. > > Now, after gathering some experience running it, we are (as I'm writing > > this, gathered in the same room) working on an improved proposal for > > KIP-1150 that also addresses the concerns from the community. > > We expect to share the updated KIP in the next couple of weeks. > > > > We apologize for the recent period of silence and are committed to more > > regular communication as we move forward. > > > > Best, > > > > > > On Tue, Aug 5, 2025 at 10:31 AM Luke Chen <show...@gmail.com> wrote: > > > > > Hi all, > > > > > > The Kafka community is currently seeing an unprecedented situation with > > > three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the > same > > > challenge of high replication costs when running Kafka across multiple > > > cloud availability zones. Each KIP offers a different solution to this > > > issue. While diversity of innovative ideas is a key strength of > open-source > > > projects, it creates a burden for reviewers and users who must compare > and > > > comment on multiple proposals simultaneously. Furthermore, discussion > > > around the three KIPs has stalled for over two months now. This could > be > > > due to the authors being hesitant to proceed due to the existence of > > > alternative, potentially conflicting, solutions. Addressing replication > > > cost is a key concern of Kafka’s userbase and we should try to move the > > > conversation forward if we can. > > > > > > From what I understand, these three KIPs are not mutually exclusive. > But > > > adopting all three KIPs in the community might not be what we expect. > Thus, > > > I would like to *start a discussion on how we could move the > conversation > > > forward*. > > > > > > To save time for the KIP readers/reviewers, I have created this > document > > > < > > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > > >[1] > > > to help summarize each of the KIPs and describe their current status. > *Hope > > > to get some suggestions/feedback from the community*. > > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > > > > > KIP-1150: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > > KIP-1176 > > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+TopicsKIP-1176 > > > > > : > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+Segment > > > KIP-1183 > > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+SegmentKIP-1183 > > > > > : > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1183%3A+Unified+Shared+Storage > > > > > > > > > Thank you. > > > Luke > > > > > > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Sr. Engineering Director, Streaming Services, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > > > Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
