Thanks for confirmation. If any inputs from my side, definitely will come back share.
From: Hongshun Wang <[email protected]> Sent: 30 July 2025 14:35 To: Frens Jan Rumph <[email protected]> Cc: Kamal Mittal <[email protected]>; [email protected] Subject: Re: Kafka partition assignment for consumers Hi Kamal, > Does Flink assign partitions to consumers for Kafka Source or Kafka itself > does this? Flink does it. > may be if you can give inputs that how flink handles consumers scaling? I am working on FLIP-537[1], and also describe how the flink kafka connector assigns splits and currently problem. Maybe you can refer to it and give me some advice. Best, Hongshun [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-537%3A+Enumerator+with+Global+Split+Assignment+Distribution+for+Balanced+Split+assignment?src=contextnavpagetreemode On Fri, Jul 25, 2025 at 4:07 AM Frens Jan Rumph <[email protected]<mailto:[email protected]>> wrote: No problem. As I said, I’m no expert in this domain myself, but from what I’ve read I have no reason to think you should expect any fundamental issues. I don’t know what you should expect in terms of failover latency and such. I expect that under normal circumstances (maintenance, deliberate scaling etc) this should be fairly minimal. Also here obviously depending on what availability and latency is required by your use case. I’m not sure how partition redistribution is handled in Flink in detail. I’m not aware of any load balancing mechanism or other rebalancing that is triggered. The docs state that partitions are distributed - as they are discovered - to readers (task) in a round robin fashion. Take a look at addReader in KafkaSourceEnumerator, that should give you some further details. I’m not even sure if you even need something like Cruise control. That’s really something that depends on the scale of operations you foresee I guess. Heck, at LinkedIn scale, apparently even Cruise control doesn’t even cut it? - given their announcement of NorthGuard<https://lnkd.in/gTYjMncW>. I’m a little out of my depth here, I can only think along so far. Best regards, Frens On 24 Jul 2025, at 16:45, Kamal Mittal <[email protected]<mailto:[email protected]>> wrote: Thanks for the details. So query is more around "rebalancing" which is done by kafka itself in case of consumer scaling and by cruise control incase of broker scaling. During consumer scaling, partition redistribution is handled by flink without causing any issue with partition splits and consumer offsets? Also as redistribution might take some time depending upon partition size so flink kafka consumer may need to hold for read? In this case kafka is given responsibility to assign partitions as application only gives topics configuration to flink kafka source. During broker scaling, partition (replica) movement happens and, in that case, flink kafka source and sink will be able to handle this replica movement? Let me know please if misunderstood something. ________________________________ Hi Kamal, That’s what you hope for always; that people first search through documentation. That people search flink code is maybe even beyond that, nice! In this case I’m no expert. That said, I think the Behind the Scene section on https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene provides some important information: The split enumerator of Kafka is responsible for discovering new splits (partitions) under the provided topic partition subscription pattern, and assigning splits to readers, uniformly distributed across subtasks, in round-robin style. Note that the split enumerator of Kafka source pushes splits eagerly to source readers, so it won’t need to handle split requests from source reader. The code relevant to assigning partitions should be available in https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java. So, in short, Flink seems to organises (assigns) the splits (~ the Flink counterpart of Kafka partitions) over available source readers (in tasks in task executors). As for your question about Cruise control for kafka cluster rebalancing, I think you have things mixed up here. I didn’t know about this LinkedIn project, but apparently it (re)balances partitions over brokers. This somewhat relates to, but is not the same as balancing partitions over consumers. Looping the user mailing list back in as others may chip in or tell me I’m completely wrong ;) As I said, I’m a Flink user, and I’ve worked with Kafka, but I’m far from an expert on the matter. Don’t treat my answer as authoritative in anyway! Best regards, Frens Jan On 24 Jul 2025, at 13:32, Kamal Mittal <[email protected]<mailto:[email protected]>> wrote: Sure, truly speaking I myself first read the documentation I found and even searched the flink code for my query but didn’t find anything concrete. I am trying to introduce cruise control for kafka cluster rebalancing and due to that only I asked this query. I even asked about “kafka cluster rebalancing” for which yet to receive any feedback. From: Frens Jan Rumph <[email protected]<mailto:[email protected]>> Sent: 24 July 2025 16:41 To: Kamal Mittal <[email protected]<mailto:[email protected]>> Subject: Re: Kafka partition assignment for consumers Hi Kamal, Thanks for getting back to me! Good to get this conversation starting. I realise that tone of voice is subjective. There’s no need to be overly polite, but - at least in my view - some decorum is often helpful. Best regards, Frens Jan On 24 Jul 2025, at 13:00, Kamal Mittal <[email protected]<mailto:[email protected]>> wrote: Dear Frens, Thanks for your message. I’m fully aware that this is a community mailing list, and I’ve always approached it with the intent of constructive discussion and knowledge sharing. Asking for input on technical questions — even briefly — is not out of place in such forums. If every message had to meet arbitrary length or formality standards, participation would likely drop, not improve. As for the line you quoted — “Need inputs for below query. Please share views.” — I don’t see how that’s "unacceptable." It was a concise and polite request, not a demand. If the brevity offended you, that wasn’t the intent — but it’s worth recognizing that tone interpretation is subjective, especially in written communication. Lastly, suggesting that someone is treating the list like a "help desk" just because they asked a question feels unnecessarily harsh. Communities thrive when people feel welcome to engage, not when they’re policed for wording. That said, I’ll certainly continue to be mindful of how my messages are framed. Best, Kamal From: Frens Jan Rumph <[email protected]<mailto:[email protected]>> Sent: 24 July 2025 10:25 To: Kamal Mittal <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: Kafka partition assignment for consumers Dear Kamal, I am not an active member of the Flink community, let alone a moderator or admin; so I’m probably stepping out of line here. And I’m going to be a little bit blunt here, but I don’t think you should treat this mailing list as a help desk. You need to realise that this is a community mailing list. Your e-mail ends up in the mailbox of probably hundreds or maybe even thousands of people. I think that you should work on your tone of voice. Something like this is at least to me unacceptable: > Need inputs for below query. Please share views. Also, I think you should improve your questions themselves. You should indicate what do you already know, what you did to answer the question yourself and what documentation was missing. In the case of how Flink handles Kafka partition assignment, have you read the documentation on https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene? It even links to some great additional resources. Best regards, Frens Jan On 24 Jul 2025, at 06:04, Kamal Mittal via user <[email protected]<mailto:[email protected]>> wrote: OR may be if you can give inputs that how flink handles consumers scaling? From: Kamal Mittal via user <[email protected]<mailto:[email protected]>> Sent: 24 July 2025 09:16 To: [email protected]<mailto:[email protected]> Subject: Kafka partition assignment for consumers Hello, Does Flink assign partitions to consumers for Kafka Source or Kafka itself does this? Rgds, Kamal
