Chia-Ping,
Thanks for looking into our proposal (KIP-1176). We have a preliminary
prototype running in our cluster, there are still some bugs and enhancements we
want to make to make it more mature. Given there are multiple KIPs proposed in
this area, we want to share our proposal to the community early to get some
feedbacks.
On Tuesday, May 6, 2025 at 01:59:37 AM PDT, Chia-Ping Tsai
<chia7...@apache.org> wrote:
hi Henry Haiying Cai
Thanks for sharing the cool idea. Out of curiosity, do you already have an
implementation running in your cluster?
Best,
Chia-Ping
On 2025/05/06 05:56:28 Henry Haiying Cai wrote:
> If it's not too late into the party, we (Slack/Salesforce) also have
>submitted KIP-1176: Tiered Storage for Active Log Segments . Our proposal is
>not a diskless offering but an incremental evolution on top of KIP-405: Tiered
>Storage. We propose using background tasks to upload a section of active log
>segments from the leader broker to the fast object storage such as S3E1Z and
>download them onto the follower broker. As a result the data flows from the
>leader broker to the object storage and then to the follower broker without
>paying for across-AZ transfer cost.
> Comparing to KIP-1150 Diskless Topics, we have a much simpler design by
> reusing most of the constructs from KIP-405: tiered storage. By stilling
> using local log segment files and Kafka's core performance tenet: Page Cache,
> we are able to maintain the same single digit millisecond performance for
> acks=1 (and close performance on acks=-1) while cutting overall cost by 43%
> through cutting across-AZ data transfer. (Comparing to KIP-1150 which has a
> much longer latency in the order of 100ms since the producer acknowledgement
> needs to wait for object storage upload to finish).
> On Thursday, April 17, 2025 at 04:11:43 AM PDT, Luke Chen
><show...@gmail.com> wrote:
>
> Hi Josep,
>
> Thanks for the KIP!
> Quite exciting to see this feature brought into Apache Kafka!!!!
>
> Comments:
> 1. "Permit multi-region active-active topics with automatic failover"
> I didn't see any future work mentioning this. Does it mean, with diskless
> topic MVP, this will work by default?
>
> 2. "Write through to object storage, avoiding local disk usage"
> While this title and the goal said no local disk usage, I'd like to make
> sure is it really zero local disk needed?
> We might need to clarify it in the KIP.
>
> Thank you.
> Luke
>
> On Wed, Apr 16, 2025 at 7:58 PM Josep Prat <josep.p...@aiven.io.invalid>
> wrote:
>
> > Hi Kafka Devs!
> >
> > We want to start a new KIP discussion about introducing a new type of
> > topics that would make use of Object Storage as the primary source of
> > storage. However, as this KIP is big we decided to split it into multiple
> > related KIPs.
> > We have the motivational KIP-1150 (
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
> > )
> > that aims to discuss if Apache Kafka should aim to have this type of
> > feature at all. This KIP doesn't go onto details on how to implement it.
> > This follows the same approach used when we discussed KRaft.
> >
> > But as we know that it is sometimes really hard to discuss on that meta
> > level, we also created several sub-kips (linked in KIP-1150) that offer an
> > implementation of this feature.
> >
> > We kindly ask you to use the proper DISCUSS threads for each type of
> > concern and keep this one to discuss whether Apache Kafka wants to have
> > this feature or not.
> >
> > Thanks in advance on behalf of all the authors of this KIP.
> >
> > ------------------
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io | +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
