Re: FlinkKafkaConsumer configuration to consume from Multiple Kafka Topics

2018-07-18 Thread Andrey Zagrebin
Hi Sagar, At the moment number of partitions in Kafka source topics and parallelism of Flink Kafka source operator are completely independent. Flink will internally distribute partitions between a number of source parallel subtasks which you configure. In case of dynamic partition or topic disc

FlinkKafkaConsumer configuration to consume from Multiple Kafka Topics

2018-07-17 Thread sagar loke
Hi, We have a use case where we are consuming from more than 100s of Kafka Topics. Each topic has different number of partitions. As per the documentation, to parallelize a Kafka Topic, we need to use setParallelism() == number of Kafka Partitions for a topic. But if we are consuming multiple to