Thanks Jing for starting the discussion. +1 for removing FlinkKafkaConsumer, as KafkaSource has evolved for many release cycles and should be stable enough. I have some concerns about the new Kafka sink based on sink v2, as sink v2 still has some ongoing work in 1.17 (maybe Yun Gao could provide some inputs). Also we found some issues of KafkaSink related to the internal mechanism of sink v2, like FLINK-29492.
@David About the ability of DeserializationSchema#isEndOfStream, FLIP-208 is trying to complete this piece of the puzzle, and Hang Ruan ( ruanhang1...@gmail.com) plans to work on it in 1.17. For the partition idleness problem could you elaborate more about it? I assume both FlinkKafkaConsumer and KafkaSource need a WatermarkStrategy to decide whether to mark the partition as idle. Best, Qingsheng Ververica (Alibaba) On Thu, Oct 27, 2022 at 8:06 PM Jing Ge <j...@ververica.com> wrote: > Hi Dev, > > I'd like to start a discussion about removing FlinkKafkaConsumer and > FlinkKafkaProducer in 1.17. > > Back in the past, it was originally announced to remove it with Flink 1.15 > after Flink 1.14 had been released[1]. And then postponed to the next 1.15 > release which meant to remove it with Flink 1.16 but forgot to change the > doc[2]. I have created a PRs to fix it. Since the 1.16 release branch has > code freeze, it makes sense to, first of all, update the doc to say that > FlinkKafkaConsumer will be removed with Flink 1.17 [3][4] and second start > the discussion about removing them with the current master branch i.e. for > the coming 1.17 release. I'm all ears and looking forward to your feedback. > Thanks! > > Best regards, > Jing > > > > > > > > > > [1] > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#kafka-sourcefunction > [2] > > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#kafka-sourcefunction > [3] https://github.com/apache/flink/pull/21172 > [4] https://github.com/apache/flink/pull/21171 >
