Hi Kamal, > Does Flink assign partitions to consumers for Kafka Source or Kafka itself does this?
Flink does it. > may be if you can give inputs that how flink handles consumers scaling? I am working on FLIP-537[1], and also describe how the flink kafka connector assigns splits and currently problem. Maybe you can refer to it and give me some advice. Best, Hongshun [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-537%3A+Enumerator+with+Global+Split+Assignment+Distribution+for+Balanced+Split+assignment?src=contextnavpagetreemode On Fri, Jul 25, 2025 at 4:07 AM Frens Jan Rumph <[email protected]> wrote: > No problem. As I said, I’m no expert in this domain myself, but from what > I’ve read I have no reason to think you should expect any fundamental > issues. I don’t know what you should expect in terms of failover latency > and such. I expect that under normal circumstances (maintenance, deliberate > scaling etc) this should be fairly minimal. Also here obviously depending > on what availability and latency is required by your use case. > > I’m not sure how partition redistribution is handled in Flink in detail. > I’m not aware of any load balancing mechanism or other rebalancing that is > triggered. The docs state that partitions are distributed - as they are > discovered - to readers (task) in a round robin fashion. Take a look > at addReader in KafkaSourceEnumerator, that should give you some further > details. > > I’m not even sure if you even need something like Cruise control. That’s > really something that depends on the scale of operations you foresee I > guess. Heck, at LinkedIn scale, apparently even Cruise control doesn’t even > cut it? - given their announcement of NorthGuard > <https://lnkd.in/gTYjMncW>. > > I’m a little out of my depth here, I can only think along so far. > > Best regards, > Frens > > > > On 24 Jul 2025, at 16:45, Kamal Mittal <[email protected]> wrote: > > Thanks for the details. > > So query is more around "rebalancing" which is done by kafka itself in > case of consumer scaling and by cruise control incase of broker scaling. > > During consumer scaling, partition redistribution is handled by flink > without causing any issue with partition splits and consumer offsets? Also > as redistribution might take some time depending upon partition size so > flink kafka consumer may need to hold for read? In this case kafka is given > responsibility to assign partitions as application only gives topics > configuration to flink kafka source. > > During broker scaling, partition (replica) movement happens and, in that > case, flink kafka source and sink will be able to handle this replica > movement? > > Let me know please if misunderstood something. > > ------------------------------ > > Hi Kamal, > > That’s what you hope for always; that people first search through > documentation. That people search flink code is maybe even beyond that, > nice! > > > In this case I’m no expert. > > That said, I think the Behind the Scene section on > https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene > provides > some important information: > > The split enumerator of Kafka is responsible for discovering new splits > (partitions) under the provided topic partition subscription pattern, and > assigning splits to readers, uniformly distributed across subtasks, in > round-robin style. Note that the split enumerator of Kafka source pushes > splits eagerly to source readers, so it won’t need to handle split requests > from source reader. > > > The code relevant to assigning partitions should be available in > https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java > . > > > So, in short, Flink seems to organises (assigns) the splits (~ the Flink > counterpart of Kafka partitions) over available source readers (in tasks in > task executors). > > > As for your question about *Cruise control for kafka cluster rebalancing*, > I think you have things mixed up here. I didn’t know about this LinkedIn > project, but apparently it (re)balances partitions over *brokers*. This > somewhat relates to, but is not the same as balancing partitions over > consumers. > > > Looping the user mailing list back in as others may chip in or tell me I’m > completely wrong ;) As I said, I’m a Flink user, and I’ve worked with > Kafka, but I’m far from an expert on the matter. Don’t treat my answer as > authoritative in anyway! > > Best regards, > Frens Jan > > > > > On 24 Jul 2025, at 13:32, Kamal Mittal <[email protected]> wrote: > > Sure, truly speaking I myself first read the documentation I found and > even searched the flink code for my query but didn’t find anything concrete. > > I am trying to introduce cruise control for kafka cluster rebalancing and > due to that only I asked this query. I even asked about “kafka cluster > rebalancing” for which yet to receive any feedback. > > *From:* Frens Jan Rumph <*[email protected] <[email protected]>*> > *Sent:* 24 July 2025 16:41 > *To:* Kamal Mittal <*[email protected] > <[email protected]>*> > *Subject:* Re: Kafka partition assignment for consumers > > Hi Kamal, > > Thanks for getting back to me! Good to get this conversation starting. I > realise that tone of voice is subjective. There’s no need to be overly > polite, but - at least in my view - some decorum is often helpful. > > Best regards, > Frens Jan > > > > On 24 Jul 2025, at 13:00, Kamal Mittal <*[email protected] > <[email protected]>*> wrote: > > Dear Frens, > > Thanks for your message. > > I’m fully aware that this is a community mailing list, and I’ve always > approached it with the intent of constructive discussion and knowledge > sharing. Asking for input on technical questions — even briefly — is not > out of place in such forums. If every message had to meet arbitrary length > or formality standards, participation would likely drop, not improve. > > As for the line you quoted — “Need inputs for below query. Please share > views.” — I don’t see how that’s "unacceptable." It was a concise and > polite request, not a demand. If the brevity offended you, that wasn’t the > intent — but it’s worth recognizing that tone interpretation is subjective, > especially in written communication. > > Lastly, suggesting that someone is treating the list like a "help desk" > just because they asked a question feels unnecessarily harsh. Communities > thrive when people feel welcome to engage, not when they’re policed for > wording. > > That said, I’ll certainly continue to be mindful of how my messages are > framed. > > Best, > Kamal > > *From:* Frens Jan Rumph <*[email protected] <[email protected]>*> > *Sent:* 24 July 2025 10:25 > *To:* Kamal Mittal <*[email protected] > <[email protected]>*> > *Cc:* *[email protected] <[email protected]>* > *Subject:* Re: Kafka partition assignment for consumers > > Dear Kamal, > > I am not an active member of the Flink community, let alone a moderator or > admin; so I’m probably stepping out of line here. > > And I’m going to be a little bit blunt here, but I don’t think you should > treat this mailing list as a help desk. You need to realise that this is a > community mailing list. Your e-mail ends up in the mailbox of probably > hundreds or maybe even thousands of people. > > I think that you should work on your tone of voice. Something like this is > at least to me unacceptable: > > Need inputs for below query. Please share views. > > Also, I think you should improve your questions themselves. You should > indicate what do you already know, what you did to answer the question > yourself and what documentation was missing. > > In the case of how Flink handles Kafka partition assignment, have you read > the documentation on > *https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene? > <https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/#behind-the-scene?>* > It > even links to some great additional resources. > > Best regards, > Frens Jan > > > > > > > On 24 Jul 2025, at 06:04, Kamal Mittal via user <*[email protected] > <[email protected]>*> wrote: > > OR may be if you can give inputs that how flink handles consumers scaling? > > *From:* Kamal Mittal via user <*[email protected] > <[email protected]>*> > *Sent:* 24 July 2025 09:16 > *To:* *[email protected] <[email protected]>* > *Subject:* Kafka partition assignment for consumers > > Hello, > > Does Flink assign partitions to consumers for Kafka Source or Kafka itself > does this? > > Rgds, > Kamal > > > > >
