Hi, Jan,
For me, the main gap of KIP-1150 is the support of all existing client
APIs. Currently, there is no design for supporting APIs like transactions
and queues.
Thanks,
Jun
On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski
wrote:
> Would it be a good time to ask for the current status of th
Would it be a good time to ask for the current status of this KIP? I
haven't seen much activity here for the past 2 months, the vote got vetoed
but I think the pending questions have been answered since then. KIP-1183
(AutoMQ's proposal) also didn't have any activity since May.
In my eyes KIP-1150
I have some nits - it may be useful to
a) group all the KIP email threads in the main one (just a bunch of links to
everything)
b) create the email threads
It's a bit hard to track it all - for example, I was searching for a discuss
thread for KIP-1165 for a while; As far as I can tell, it does
Hi Jun,
Thanks for the great feedback. Here are the team responses:
> JR1. Sounds good. It would be useful to document the benefit from each
major Cloud in the KIP.
Sure. We are adding a reference to the current cross-AZ charges by the
major providers.
> JR2.
We are working on validating the s
Hi, Tao Jiuming.
The Diskless proposal doesn’t plan to modify the Kafka Storage API. I'm
adding a note about this in the KIP.
Please, see KIP-1163 for more details about the write path. We can also
follow-up the discussion in its own discussion thread.
Thanks,
Jorge.
On Thu, 15 May 2025 at 12:5
Thanks Colin! As one of the co-authors, here are the team responses:
> I think there's a bit of confusion in the motivation and naming here. As
Jun said, what's being proposed here is not truly "diskless" -- we're still
storing a fair amount of metadata on local disks.
We propose to address the n
Hi, Giuseppe, Ivan,
Thanks for the reply.
JR1. Sounds good. It would be useful to document the benefit from each
major Cloud in the KIP.
JR2.
"The same tiered storage doesn’t support compacted topics and the
community seems fine with this, because compacted topics are not often that
big to benef
On Tue, May 13, 2025, at 19:34, Jun Rao wrote:
>
> JR4. "Balance traffic among brokers and eliminate broker hotspots with
> per-client granularity". Does that mean all traffic from a client is served
> from a single broker? This seems to reduce the scalability from the client
> perspective.
We p
On Tue, May 13, 2025, at 19:34, Jun Rao wrote:
> JR3. "Permit multi-region active-active topics with automatic failover".
> Could you elaborate on the benefit of this? Cloud providers still charge
> cross region data transfer in object stores, right?
First, we should have emphasized in the KIP th
Hi Jun,
Thank you for your questions!
On Tue, May 13, 2025, at 19:34, Jun Rao wrote:
> Hi, Josep,
>
> Thanks for the KIP. At the highlevel, the KIP is well thought through and
> provides multiple benefits for Kafka in the Cloud. A few comments below.
>
> JR1. One of the key motivations is to el
Hello Jun,
Thank you for your interest in the KIP.
Regarding your question about transactions:
> JR2. Transactions on Diskless Topics is listed in the future work.
> Currently, we try to support all existing client APIs for every new
> feature. For example, remote storage (KIP-405) supports tran
looks like it will write data to remote OSS directly,
I'm just wandering that do we need to make storage API async?
Kafka's storage API is fully sync, in this case, if we write data to s3
directly and wait the response, I'm afraid the throughput can be very low.
But if we make it async, it will
Hi Josep,
Thanks for the KIP.
I think there's a bit of confusion in the motivation and naming here. As Jun
said, what's being proposed here is not truly "diskless" -- we're still storing
a fair amount of metadata on local disks.
The proposal talks about "Unification/Relationship with Tiered St
Hi, Josep,
Thanks for the KIP. At the highlevel, the KIP is well thought through and
provides multiple benefits for Kafka in the Cloud. A few comments below.
JR1. One of the key motivations is to eliminate inter-zone data transfer
costs from Kafka replication. It would be useful to provide a shor
Hi Jan,
Thanks for the question and interest in the KIP. I'll be participating in
this discussion from the authors' side as well.
KIP-E has been published today as
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1181%3A+Metadata+Rack+Awareness+for+Diskless+Topics
and the proposal there is a
Hi Ivan,
KIP-E: Producer Rack Awareness seems to be a subset of KIP-1123. I don’t know
the process but maybe it’s worth linking somehow?
(I hope this reply lands under the correct message, I haven’t used mailing
lists in a decade or two)
Kind regards,
Jan
On 2025/04/23 12:04:45 Ivan Yurchenko
Chia-Ping,
Thanks for looking into our proposal (KIP-1176). We have a preliminary
prototype running in our cluster, there are still some bugs and enhancements we
want to make to make it more mature. Given there are multiple KIPs proposed in
this area, we want to share our proposal to the comm
hi Henry Haiying Cai
Thanks for sharing the cool idea. Out of curiosity, do you already have an
implementation running in your cluster?
Best,
Chia-Ping
On 2025/05/06 05:56:28 Henry Haiying Cai wrote:
> If it's not too late into the party, we (Slack/Salesforce) also have
> submitted KIP-1176:
If it's not too late into the party, we (Slack/Salesforce) also have submitted
KIP-1176: Tiered Storage for Active Log Segments . Our proposal is not a
diskless offering but an incremental evolution on top of KIP-405: Tiered
Storage. We propose using background tasks to upload a section of ac
Hello Kafka devs!
I finally found some time to sit down to this topic, I hope I'm not too late to
the party with my thoughts ;)
I like the proposed solution a lot, how it's thoroughly explained and thought
through. The essence of it does more or less what Bufstream and Warpstream are
doing and
Dear Luke,
Your suggestion to write a KIP for this proposal is excellent.
I will be submitting a KIP within the next week or two, and will start
a separate thread for discussion.
Best regards,
Xiaorui
On Thu, Apr 24, 2025 at 3:10 PM Luke Chen wrote:
>
> Hi Xiaorui,
>
> It's great to hear that
Hi Xiaorui,
It's great to hear that the AutoMQ team is also interested in contributing
to Apache Kafka.
Of course the AutoMQ's solution is also an option for the community to
consider.
It would be good if you can write a KIP for your solution, and we can
discuss it in the community.
Let me know i
gt; Then, how Batch Coordinator can choose totol order to be A,B,C ?
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -
> 发件人: "Christo Lolov"
> 收件人: dev@kafka.apache.org
> 发送时间: 星期二, 2025年 4 月 22日 下午 9:04:06
> 主题: [SPAM]Re: [DISCUSS] KIP-1150 Diskless Topics
&
Hi Christo and all!
Thank you for your questions.
> CL - 1: In the same lane as Luke's comment, it would be very useful to see
> explicitly what will stay on disk and what won't stay on disk
Good point. I added a small section "Disk usage and lack thereof" to KIP-1150.
> CL - 2: It would also b
Dear Community,
I am truly delighted to see the KIP-1150 proposal put forth by Josep, which
presents an exciting new architecture for the cloud economy.
It was an honor to hear Ziming and Stanislav mention AutoMQ[1] during the
discussion. Over the past few years, AutoMQ has dedicated itself to
.apache.org
发送时间: 星期二, 2025年 4 月 22日 下午 9:04:06
主题: [SPAM]Re: [DISCUSS] KIP-1150 Diskless Topics
Hello!
I want to start with saying that this is a big and impressive undertaking
and I am really excited to see its progression! I am posting my initial
comments in this thread, but they span a few
Hello!
I want to start with saying that this is a big and impressive undertaking
and I am really excited to see its progression! I am posting my initial
comments in this thread, but they span a few of the child KIPs. Let me know
which questions you would like to move elsewhere. I understand that y
This is an amazing initiative. Huge kudos for driving it. We should incorporate
it one way or another.
I have a suggestion I'd like to hear your thoughts on. I'm cognizant of the
effort required for KIP-1150 so I don't necessarily want to increase the scope
- but thinking about this early on ca
Hi Ziming,
> 1. Is this feature available by just a minor adjust of config or it will
> intrude current code heavily, say, AutoMq is 100% compatible with Kafka and
> doesn’t intrude the code heavily
If we speak about the part visible to the user, we expect:
1. Minimal changes to the client co
Hi Luke and all!
I'll be participating in this discussion from the authors' side together with
Josep and some other colleagues.
> 2. "Write through to object storage, avoiding local disk usage"
> While this title and the goal said no local disk usage, I'd like to make
> sure is it really zero lo
Hi Josep,
This would be a fascinating feature, some well known Kafka users are using
Kafka in a cloud-native env. As for as I know, there are already some secondary
development version Kafka which provide this feature, for example, I am using
AutoMq(https://github.com/AutoMQ/automq) in my envir
Hi Josep,
Thanks for the KIP!
Quite exciting to see this feature brought into Apache Kafka
Comments:
1. "Permit multi-region active-active topics with automatic failover"
I didn't see any future work mentioning this. Does it mean, with diskless
topic MVP, this will work by default?
2. "Write
32 matches
Mail list logo
