Hi Prasanna, which Flink version and Kafka connector are you using? (the "KafkaSource" or "FlinkKafkaConsumer"?)
The partition assignment for the FlinkKafkaConsumer is defined here: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/KafkaTopicPartitionAssigner.java#L27-L43 I assume all your topics have one partition only. Still, the "startIndex" should be determined based on the hash of the topic name. My only explanation is that your unlucky with the distribution of the hashes. If this leads to performance issues, consider using topics with multiple partitions, change the name of the topics or increase the parallelism of your consumer. On Tue, Jul 20, 2021 at 7:53 AM Prasanna kumar < prasannakumarram...@gmail.com> wrote: > Hi, > > We have a Flink job reading from multiple Kafka topics based on a regex > pattern. > > What we have found out is that the topics are not shared between the kafka > consumers in an even manner . > > Example if there are 8 topics and 4 kafka consumer operators . 1 > consumer is assigned 6 topics , 2 consumers assigned 1 each and the last > consumer is not assigned at all. > > This leads to inadequate usage of the resources. > > I could not find any setting/configuration which would make them as even > as possible. > > Let me know if there's a way to do the same. > > Thanks, > Prasanna. >
