Hi Ivan, Thanks for the KIP. The Jira link for this KIP is KAFKA-1. Could you update to a correct Jira number?
Best, PoAn > On Mar 30, 2025, at 2:13 AM, Ivan Yurchenko <i...@ivanyu.me> wrote: > > Hi Federico, > > There's a sentence about reduced latency in the Motivation section. Do you > think that'd be enough? > > Best, > Ivan > > > On Tue, Mar 25, 2025, at 09:23, Federico Valeri wrote: >> Hi, thanks for the KIP and sorry for the late reply. Should we also >> highlight the reduced latency in the motivation section? >> >> >> On Mon, Mar 10, 2025 at 9:24 PM Ivan Yurchenko <i...@ivanyu.me> wrote: >>> >>> Hi all, >>> >>> I took the pause with this KIP while Kafka 4.0 was in making to not >>> distract the folks. Now let's continue the discussion! >>> >>> Thank you for the comments, Luke! I've applied your suggestions. >>> >>> Best, >>> Ivan >>> >>> On Mon, Dec 23, 2024, at 03:23, Luke Chen wrote: >>>> Hi Ivan, >>>> >>>> Thanks for the KIP! >>>> This is a great improvement from the cost and latency perspective! >>>> >>>> Some comments: >>>> 1. In the description of `partitioner.rack.aware` config, it'd be better to >>>> make it clear that this setting has no effect if a custom partitioner is >>>> used. >>>> >>>> 2. "Select the next partition from all partitions following the current >>>> algorithm in the following cases:" >>>> I think there should be one more case that "If the "partitioner.rack.aware" >>>> is false; >>>> >>>> 3. "If the automatic partitioning is needed (i.e. no record partition or >>>> key is specified):" >>>> I think we should also add the case: "key is provided but >>>> `partitioner.ignore.keys` >>>> is enabled" >>>> >>>> Thank you. >>>> Luke >>>> >>>> >>>> On Sat, Dec 21, 2024 at 2:32 AM Stanislav Kozlovski < >>>> stanislavkozlov...@apache.org> wrote: >>>> >>>>> Wow, I am super happy to see this KIP! Thanks for publishing it! >>>>> >>>>> I threw the idea out there last week in an article of mine about >>>>> calculating Kafka costs[1] >>>>> >>>>>> [FUTURE KIP] - a Produce to Local Leader KIP, similar to KIP-392, can be >>>>> introduced to eliminate producer inter-AZ network costs for topics that do >>>>> not have keys. >>>>>> there is no fundamental reason that a topic without ordering guarantees >>>>> needs to produce to a specific partition - why not just choose the broker >>>>> in the closest zone? >>>>>> if all of your traffic is unkeyed, then this can further reduce Kafka’s >>>>> network cost by 25%. >>>>>> it sounds like a change that wouldn’t be too complicated, maybe even >>>>> achievable today through the Producer’s partitioner. >>>>> >>>>> I don't know if you saw it from there, but I'm super happy to see it come >>>>> to fruition! It's even easier than I thought - I didn't realize we had the >>>>> node/rack information in the partitioner already. >>>>> >>>>> I think it will be very impactful. >>>>> We've seen the strong trend in the industry of trading off latency for >>>>> cost reduction. Namely - almost every vendor has introduced some sort of >>>>> leaderless Kafka API model that outsources replication to a remote store >>>>> cost[2][3][4][5]. This in turn allows them to reduce cross-zone networking >>>>> costs to literally zero. In certain optimized deployments the networking >>>>> cost can be up to 80-90% of the total cost![6] KIP-392 allows us to >>>>> eliminate the consumer-side traffic cost, but there is great motivation to >>>>> enable users to do the same for producers that don't depend on ordering. >>>>> >>>>> I am +1 the KIP as is. >>>>> >>>>> One may make an argument to have a way to enable it server-side via the >>>>> broker, but I'd like to hear a good reason for that. I believe the >>>>> simplicity in the current state is preferred, since clients already have >>>>> freedom to produce to any partition they explicitly choose. >>>>> >>>>> Best, >>>>> Stan >>>>> >>>>> [1] >>>>> https://bigdata.2minutestreaming.com/p/the-brutal-truth-about-apache-kafka-cost-calculators >>>>> [2] WarpStream and its $220m acquisition >>>>> https://www.linkedin.com/pulse/how-confluent-acquired-warpstream-220m-after-just-13-months-hxgyf/ >>>>> [3] Confluent Freight >>>>> https://www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/ >>>>> [4] RedPanda Cloud Topics >>>>> https://www.redpanda.com/blog/cloud-topics-streaming-data-object-storage >>>>> [5] BufStream https://buf.build/product/bufstream >>>>> [6] calculator https://akalculator.com/ >>>>> >>>>> On 2024/12/20 11:35:28 Ivan Yurchenko wrote: >>>>>> Hello all, >>>>>> >>>>>> I'd like to propose a new KIP to discuss: KIP-1123: Rack-aware >>>>> partitioning for Kafka Producer [1]. >>>>>> >>>>>> Best, >>>>>> Ivan Yurchenko >>>>>> >>>>>> [1] >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123%3A+Rack-aware+partitioning+for+Kafka+Producer >>>>>> >>>>> >>>> >>
