Hi Federico, There's a sentence about reduced latency in the Motivation section. Do you think that'd be enough?
Best, Ivan On Tue, Mar 25, 2025, at 09:23, Federico Valeri wrote: > Hi, thanks for the KIP and sorry for the late reply. Should we also > highlight the reduced latency in the motivation section? > > > On Mon, Mar 10, 2025 at 9:24 PM Ivan Yurchenko <i...@ivanyu.me> wrote: > > > > Hi all, > > > > I took the pause with this KIP while Kafka 4.0 was in making to not > > distract the folks. Now let's continue the discussion! > > > > Thank you for the comments, Luke! I've applied your suggestions. > > > > Best, > > Ivan > > > > On Mon, Dec 23, 2024, at 03:23, Luke Chen wrote: > > > Hi Ivan, > > > > > > Thanks for the KIP! > > > This is a great improvement from the cost and latency perspective! > > > > > > Some comments: > > > 1. In the description of `partitioner.rack.aware` config, it'd be better > > > to > > > make it clear that this setting has no effect if a custom partitioner is > > > used. > > > > > > 2. "Select the next partition from all partitions following the current > > > algorithm in the following cases:" > > > I think there should be one more case that "If the > > > "partitioner.rack.aware" > > > is false; > > > > > > 3. "If the automatic partitioning is needed (i.e. no record partition or > > > key is specified):" > > > I think we should also add the case: "key is provided but > > > `partitioner.ignore.keys` > > > is enabled" > > > > > > Thank you. > > > Luke > > > > > > > > > On Sat, Dec 21, 2024 at 2:32 AM Stanislav Kozlovski < > > > stanislavkozlov...@apache.org> wrote: > > > > > > > Wow, I am super happy to see this KIP! Thanks for publishing it! > > > > > > > > I threw the idea out there last week in an article of mine about > > > > calculating Kafka costs[1] > > > > > > > > > [FUTURE KIP] - a Produce to Local Leader KIP, similar to KIP-392, can > > > > > be > > > > introduced to eliminate producer inter-AZ network costs for topics that > > > > do > > > > not have keys. > > > > > there is no fundamental reason that a topic without ordering > > > > > guarantees > > > > needs to produce to a specific partition - why not just choose the > > > > broker > > > > in the closest zone? > > > > > if all of your traffic is unkeyed, then this can further reduce > > > > > Kafka’s > > > > network cost by 25%. > > > > > it sounds like a change that wouldn’t be too complicated, maybe even > > > > achievable today through the Producer’s partitioner. > > > > > > > > I don't know if you saw it from there, but I'm super happy to see it > > > > come > > > > to fruition! It's even easier than I thought - I didn't realize we had > > > > the > > > > node/rack information in the partitioner already. > > > > > > > > I think it will be very impactful. > > > > We've seen the strong trend in the industry of trading off latency for > > > > cost reduction. Namely - almost every vendor has introduced some sort of > > > > leaderless Kafka API model that outsources replication to a remote store > > > > cost[2][3][4][5]. This in turn allows them to reduce cross-zone > > > > networking > > > > costs to literally zero. In certain optimized deployments the networking > > > > cost can be up to 80-90% of the total cost![6] KIP-392 allows us to > > > > eliminate the consumer-side traffic cost, but there is great motivation > > > > to > > > > enable users to do the same for producers that don't depend on ordering. > > > > > > > > I am +1 the KIP as is. > > > > > > > > One may make an argument to have a way to enable it server-side via the > > > > broker, but I'd like to hear a good reason for that. I believe the > > > > simplicity in the current state is preferred, since clients already have > > > > freedom to produce to any partition they explicitly choose. > > > > > > > > Best, > > > > Stan > > > > > > > > [1] > > > > https://bigdata.2minutestreaming.com/p/the-brutal-truth-about-apache-kafka-cost-calculators > > > > [2] WarpStream and its $220m acquisition > > > > https://www.linkedin.com/pulse/how-confluent-acquired-warpstream-220m-after-just-13-months-hxgyf/ > > > > [3] Confluent Freight > > > > https://www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/ > > > > [4] RedPanda Cloud Topics > > > > https://www.redpanda.com/blog/cloud-topics-streaming-data-object-storage > > > > [5] BufStream https://buf.build/product/bufstream > > > > [6] calculator https://akalculator.com/ > > > > > > > > On 2024/12/20 11:35:28 Ivan Yurchenko wrote: > > > > > Hello all, > > > > > > > > > > I'd like to propose a new KIP to discuss: KIP-1123: Rack-aware > > > > partitioning for Kafka Producer [1]. > > > > > > > > > > Best, > > > > > Ivan Yurchenko > > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1123%3A+Rack-aware+partitioning+for+Kafka+Producer > > > > > > > > > > > > >
