Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Colt, Thanks for the feedback. > most deployments have three racks, RF=3, and one replica for each partition in each rack This KIP is mainly targeting the case where the client won't always be in the same rack as any replica. There's some proposal to make RF=2 and use other tiered storage to

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi all, I've got a rather naive question here. The spiritual goal of this KIP is to reduce cross-rack traffic (normally, this manifests itself in terms of a higher AWS/Azure bill as cloud providers charge for cross-AZ traffic). To generalize, most deployments have three racks, RF=3, and one repli

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Bruno, dropping config rack.aware.assignment.enabled and add value NONE to the enum for the possible values of config rack.aware.assignment.strategy sounds good to me. On Thu, Jun 1, 2023 at 12:39 AM Bruno Cadonna wrote: > Hi Hao, > > Thanks for the updates! > > What do you think about dropp

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Hao, Thanks for the updates! What do you think about dropping config rack.aware.assignment.enabled and add value NONE to the enum for the possible values of config rack.aware.assignment.strategy? Best, Bruno On 31.05.23 23:31, Hao Li wrote: Hi all, I've updated the KIP based on the fee

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Thanks Sophie! I've just made the updates :) On Wed, May 31, 2023 at 2:29 PM Sophie Blee-Goldman wrote: > Thanks Hao -- I really like Walker's idea to separate the assignment > strategy from the rack awareness option. > Having two different configs for these makes a lot of sense to me and seems

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi all, I've updated the KIP based on the feedback. Major changes I made: 1. Add rack aware assignment to `StickyTaskAssignor` 2. Reject `Prefer reliability and then find optimal cost` option in standby task assignment. On Wed, May 31, 2023 at 12:09 PM Hao Li wrote: > Hi all, > > Thanks for th

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Thanks Hao -- I really like Walker's idea to separate the assignment strategy from the rack awareness option. Having two different configs for these makes a lot of sense to me and seems like it will benefit the most users. Given your KIP already has a lot going on with it, I would be happy to writ

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi all, Thanks for the feedback! I will update the KIP accordingly. *For Sophie's comments:* 1 and 2. Good catch. Fixed these. 3 and 4 Yes. We can make this public config and call out the clientConsumer config users need to set. 5. It's ideal to take the previous assignment in HAAssignor into

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Hao, Most of the comments I had on this kip are already mentioned, but I did want to share my two major concerns. 1. Stability. I worry about stability. If we only have the HA assignor work with rack awareness we will have a lot of state movement in many cases. Sophie and Bruno have this conce

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Hao, Thank you for the KIP! Really interesting! In general, I think the KIP is a bit too vague. You explain the main algorithm and different options. It is not clear to me on what option we will start voting. One way out of this situation would be to cut the KIP down to the simplest optio

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hey Hao, thanks for the KIP! 1. There's a typo in the "internal.rack.aware.assignment.strategry" config, this should be internal.rack.aware.assignment.strategy. 2. > For O(E^2 * (CU)) complexity, C and U can be viewed as constant. Number of > edges E is T * N where T is the number of clients an

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Thanks John! Yeah. The ConvergenceTest looks very helpful. I will add it to the test plan. I will also add tests to verify the new optimizer will produce a balanced assignment which has no worse cross AZ cost than the existing assignor. Hao On Mon, May 22, 2023 at 3:39 PM John Roesler wrote: >

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Hao, Thanks for the KIP! Overall, I think this is a great idea. I always wanted to circle back after the Smooth Scaling KIP to put a proper optimization algorithm into place. I think this has the promise to really improve the quality of the balanced assignments we produce. Thanks for provi

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Colt, Thanks for the comments. > and I struggle to see how the algorithm isn't at least O(N) where N is the number of Tasks...? For O(E^2 * (CU)) complexity, C and U can be viewed as constant. Number of edges E is T * N where T is the number of clients and N is the number of Tasks. This is be

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hello Hao, First of all, THANK YOU for putting this together. I had been hoping someone might bring something like this forward. A few comments: **1: Runtime Complexity > Klein’s cycle canceling algorithm can solve the min-cost flow problem in O(E^2CU) time where C is max cost and U is max capaci

[DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi all, I have submitted KIP-925 to add rack awareness logic in task assignment in Kafka Streams and would like to start a discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams -- Thanks, Hao