Re: Flink does not read from some Kafka Partitions

2018-05-16 Thread Ruby Andrews
Thank you both for your responses. Looks like I may have inadvertently used 1.3.1 libraries instead of 1.4. Ruby On Wed, May 16, 2018 at 3:12 AM Tzu-Li (Gordon) Tai wrote: > Hi, > > Timo is correct - partition discovery is supported by the consumer only > starting from Flink 1.4. > > The expect

Re: Flink does not read from some Kafka Partitions

2018-05-16 Thread Tzu-Li (Gordon) Tai
Hi, Timo is correct - partition discovery is supported by the consumer only starting from Flink 1.4. The expected behaviour without partition discovery on, is that the list of partitions picked up on the first execution of the job will be the list of subscribed partition across all executions. Wh

Re: Flink does not read from some Kafka Partitions

2018-05-15 Thread Timo Walther
Hi Ruby, which Flink version are you using? When looking into the code of the org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase you can see that the behavior for using partition discovery or not depends on the Flink version. Regards, Timo Am 15.05.18 um 02:01 schrieb Ruby