hi Henry Haiying Cai Thanks for sharing the cool idea. Out of curiosity, do you already have an implementation running in your cluster?
Best, Chia-Ping On 2025/05/06 05:56:28 Henry Haiying Cai wrote: > If it's not too late into the party, we (Slack/Salesforce) also have > submitted KIP-1176: Tiered Storage for Active Log Segments . Our proposal is > not a diskless offering but an incremental evolution on top of KIP-405: > Tiered Storage. We propose using background tasks to upload a section of > active log segments from the leader broker to the fast object storage such as > S3E1Z and download them onto the follower broker. As a result the data flows > from the leader broker to the object storage and then to the follower broker > without paying for across-AZ transfer cost. > Comparing to KIP-1150 Diskless Topics, we have a much simpler design by > reusing most of the constructs from KIP-405: tiered storage. By stilling > using local log segment files and Kafka's core performance tenet: Page Cache, > we are able to maintain the same single digit millisecond performance for > acks=1 (and close performance on acks=-1) while cutting overall cost by 43% > through cutting across-AZ data transfer. (Comparing to KIP-1150 which has a > much longer latency in the order of 100ms since the producer acknowledgement > needs to wait for object storage upload to finish). > On Thursday, April 17, 2025 at 04:11:43 AM PDT, Luke Chen > <show...@gmail.com> wrote: > > Hi Josep, > > Thanks for the KIP! > Quite exciting to see this feature brought into Apache Kafka!!!! > > Comments: > 1. "Permit multi-region active-active topics with automatic failover" > I didn't see any future work mentioning this. Does it mean, with diskless > topic MVP, this will work by default? > > 2. "Write through to object storage, avoiding local disk usage" > While this title and the goal said no local disk usage, I'd like to make > sure is it really zero local disk needed? > We might need to clarify it in the KIP. > > Thank you. > Luke > > On Wed, Apr 16, 2025 at 7:58 PM Josep Prat <josep.p...@aiven.io.invalid> > wrote: > > > Hi Kafka Devs! > > > > We want to start a new KIP discussion about introducing a new type of > > topics that would make use of Object Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > ) > > that aims to discuss if Apache Kafka should aim to have this type of > > feature at all. This KIP doesn't go onto details on how to implement it. > > This follows the same approach used when we discussed KRaft. > > > > But as we know that it is sometimes really hard to discuss on that meta > > level, we also created several sub-kips (linked in KIP-1150) that offer an > > implementation of this feature. > > > > We kindly ask you to use the proper DISCUSS threads for each type of > > concern and keep this one to discuss whether Apache Kafka wants to have > > this feature or not. > > > > Thanks in advance on behalf of all the authors of this KIP. > > > > ------------------ > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
