Hi Martijn and Qingsheng, The FLIP has been updated to extend the dynamic EOF support for the PulsarSource. I have not extended this feature to other sources yet since I am not sure it is a requirement to ensure feature consistency across different sources. Could you take another look?
Thanks, Dong On Fri, Jan 7, 2022 at 11:49 PM Dong Lin <lindon...@gmail.com> wrote: > Hi Martijn, > > Thanks for the comments! In general I agree we should avoid feature > sparsity. > > In this particular case, connectors are a bit different than most other > features in Flink. AFAIK, we plan to move connectors (including Kafka and > Pulsar) out of the Flink project in the future, which means that the > development of these connectors will be mostly de-centralized (outside of > Flink) and be up to their respective maintainers. While I agree that we > should provide API/infrastructure in Flink (as this FLIP does) to support > feature consistency across connectors, I am not sure we should own the > responsibility to actually update all connectors to achieve feature > consistency, given that we don't plan to do it in Flink anyway due to its > heavy burden. > > With that being said, I am happy to follow the community guideline if we > decide that connector-related FLIP should update every connector's API to > ensure feature consistency (to a reasonable extent). For example, in this > particular case, it looks like the EOF-detection feature can be applied to > every connector (including bounded sources). Is it still sufficient to just > update Kafka, Pulsar and Kinesis? > > Thanks, > Dong > > On Tue, Jan 4, 2022 at 3:31 PM Martijn Visser <mart...@ververica.com> > wrote: > >> Hi Dong, >> >> Thanks for writing the FLIP. It focusses only on the KafkaSource, but I >> would expect that if such a functionality is desired, it should be made >> available for all unbounded sources (for example, Pulsar and Kinesis). If >> it's only available for Kafka, I see it as if we're increasing feature >> sparsity while we actually want to decrease that. What do you think? >> >> Best regards, >> >> Martijn >> >> On Tue, 4 Jan 2022 at 08:04, Dong Lin <lindon...@gmail.com> wrote: >> >> > Hi all, >> > >> > We created FLIP-208: Update KafkaSource to detect EOF based on >> > de-serialized records. Please find the KIP wiki in the link >> > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Update+KafkaSource+to+detect+EOF+based+on+de-serialized+records >> > . >> > >> > This FLIP aims to address the use-case where users need to stop a Flink >> job >> > gracefully based on the content of de-serialized records observed in the >> > KafkaSource. This feature is needed by users who currently depend on >> > KafkaDeserializationSchema::isEndOfStream() to migrate their Flink job >> from >> > FlinkKafkaConsumer to KafkaSource. >> > >> > Could you help review this FLIP when you get time? Your comments are >> > appreciated! >> > >> > Cheers, >> > Dong >> > >> >
