RE: KafkaFetcher closed before end of stream is received for all partitions.

2020-02-19 Thread Bill Wicker
user@flink.apache.org Subject: Re: KafkaFetcher closed before end of stream is received for all partitions. Hi, Your observations are correct. It is expected that the result of `KafkaDeserializationSchema#isEndOfStream` triggers a single subtask to escape its fetch loop. Therefore, if a subtask is ass

Re: KafkaFetcher closed before end of stream is received for all partitions.

2020-02-16 Thread Tzu-Li Tai
Hi, Your observations are correct. It is expected that the result of `KafkaDeserializationSchema#isEndOfStream` triggers a single subtask to escape its fetch loop. Therefore, if a subtask is assigned multiple partitions, as soon as one record (regardless of which partition it came from) signals en

KafkaFetcher closed before end of stream is received for all partitions.

2020-02-14 Thread Bill Wicker
Hello all, I'm new to Flink and I have been developing a series of POCs in preparation for a larger project that will utilize Flink. One use case we have is to utilize the same job for both batch and streaming processing using Kafka as the source. When the job is run in batch mode we expect tha