Robert
We are checking using the metric
flink_taskmanager_job_task_operator_KafkaConsumer_assigned_partitions{jobname="SPECIFICJOBNAME"}
This metric gives the number of partitions assigned to each task(kafka
consumer operator).
Prasanna.
On Wed, Aug 4, 2021 at 8:59 PM Robert Metzger wrote:
>
Hi Prasanna,
How are you checking the assignment of Kafka partitions to the consumers?
The FlinkKafkaConsumer doesn't have a rebalance() method, this is a generic
concept of the DataStream API. Is it possible that you are
somehow partitioning your data in your Flink job, and this is causing the
d
Robert
When we apply a rebalance method to the kafka consumer, it is assigning
partitions of various topics evenly.
But my only concern is that the rebalance method might have a performance
impact .
Thanks,
Prasanna.
On Wed, Aug 4, 2021 at 5:55 PM Prasanna kumar
wrote:
> Robert,
>
> Flink ve
Robert,
Flink version 1.12.2.
Flink connector Kafka Version 2..12
The partitions are assigned equally if we are reading from a single topic.
Our Use case is to read from multiple topics [topics r4 regex pattern] we
use 6 topics and 1 partition per topic for this job.
In this case , few of the k
Hi Prasanna,
which Flink version and Kafka connector are you using? (the "KafkaSource"
or "FlinkKafkaConsumer"?)
The partition assignment for the FlinkKafkaConsumer is defined here:
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/st
Hi,
We have a Flink job reading from multiple Kafka topics based on a regex
pattern.
What we have found out is that the topics are not shared between the kafka
consumers in an even manner .
Example if there are 8 topics and 4 kafka consumer operators . 1
consumer is assigned 6 topics , 2 consume
