Hi all, @Piotr When FlinkKafkaConsumer was marked as deprecated, the migration steps were included in the release notes: https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.14/#flink-24055httpsissuesapacheorgjirabrowseflink-24055 Scrolling up a bit shows how move from FlinkKafkaProducer to KafkaSink
Given the importance of the Kafka connector, I think it's smart to write a dedicated blog on how to migrate. That could also include the differences between the two implementations, as David just outlined. @Jing SinkV2 was released with Flink 1.15, so with 1.16 released there will be 2 release cycles with that API at PublicEvolving. I don't think there were breaking changes when the bug fixes were applied, because those bug fixes were also applied in a patch version (1.15.3) which can't be done unless they weren't breaking the API. Unless there will be work planned for 1.17 on the Sink, I think it can be moved to Public too. Best regards, Martijn On Tue, Nov 1, 2022 at 7:21 AM David Anderson <dander...@apache.org> wrote: > > > > [H]ow one can migrate from the > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while > > preserving exactly-once guarantees etc? > > > The responses from Fabian Paul in [1] and [2] address the question of how > to handle the migration in terms of managing the state (where the short > answer is "arrange for Kafka to be the source of truth"). > > Those threads don't get into the differences in behavior between the two > implementations. Here I'm thinking about (1) the loss of > DeserializationSchema#isEndOfStream, and the fact that you can no longer > dynamically determine when the input stream has finished, and (2) the > change to how empty partitions are handled on startup (they used to be > marked idle automatically, whereas now you must use withIdleness in the > WatermarkStrategy). > > [1] https://www.mail-archive.com/user@flink.apache.org/msg44618.html > [2] https://www.mail-archive.com/user@flink.apache.org/msg45864.html > > On Mon, Oct 31, 2022 at 7:32 PM Piotr Nowojski <pnowoj...@apache.org> > wrote: > > > Hi, > > > > Maybe a stupid question, but how one can migrate from the > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while > > preserving exactly-once guarantees etc? Is it possible? I've tried a > quick > > search and couldn't find it, but maybe I was looking in wrong places. > > > > Best, > > Piotrek > > > > pon., 31 paź 2022 o 16:40 Jing Ge <j...@ververica.com> napisał(a): > > > > > Thanks Martijn. What you said makes a lot of sense. I figure we should > do > > > it in 2 steps. > > > > > > Step 1 (with 1.17): > > > - Remove FlinkKafkaConsumer. > > > - Graduate Kafka Source from @PublicEvolving to @Public. > > > - Update doc and leave hints for customers as the reference. > > > > > > According to [1], the Kafka Sink should also be graduated with 1.17, > i.e. > > > after 1.15 and 1.16 two release cycles. But since the design change > from > > > SinkV1 to SinkV2 were significant and there were many change requests > > since > > > then, we'd better give the sink one more release cycle time to become > > more > > > stable. The other reason for giving the Sink more time is that the > > > experimental phase was only covered by one release cycle instead of two > > as > > > [1] suggested. > > > > > > Step 2 (with 1.18 ): > > > - Remove FlinkKafkaProducer. > > > - Graduate Kafka Sink from @PublicEvolving to @Public. > > > - Update doc and leave hints for customers as the reference. > > > > > > Best regards, > > > Jing > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > > > > On Thu, Oct 27, 2022 at 3:01 PM Martijn Visser < > martijnvis...@apache.org > > > > > > wrote: > > > > > > > Hi Jing, > > > > > > > > Thanks for opening the discussion. I see no issue with removing the > > > > FlinkKafkaConsumer, since it has been marked as deprecated and the > > Source > > > > API (which is used by the KafkaSource) is marked as @Public (at least > > the > > > > Base implementation) > > > > > > > > The successor of the FlinkKafkaProducer is the KafkaSink, which is > > using > > > > the Sink V2 API which is still marked as @PublicEvolving (Base > > > > implementation). I think that we should only remove the > > > FlinkKafkaProducer > > > > if we also mark the Sink V2 as @Public. I don't think that should be > a > > > > problem (since it's based on the first Sink implementation, which was > > > > Experimental in 1.14 and got replaced with Sink V2 as PublicEvolving > in > > > > 1.15). > > > > > > > > Thanks, > > > > > > > > Martijn > > > > > > > > On Thu, Oct 27, 2022 at 2:06 PM Jing Ge <j...@ververica.com> wrote: > > > > > > > > > Hi Dev, > > > > > > > > > > I'd like to start a discussion about removing FlinkKafkaConsumer > and > > > > > FlinkKafkaProducer in 1.17. > > > > > > > > > > Back in the past, it was originally announced to remove it with > Flink > > > > 1.15 > > > > > after Flink 1.14 had been released[1]. And then postponed to the > next > > > > 1.15 > > > > > release which meant to remove it with Flink 1.16 but forgot to > change > > > the > > > > > doc[2]. I have created a PRs to fix it. Since the 1.16 release > branch > > > has > > > > > code freeze, it makes sense to, first of all, update the doc to say > > > that > > > > > FlinkKafkaConsumer will be removed with Flink 1.17 [3][4] and > second > > > > start > > > > > the discussion about removing them with the current master branch > > i.e. > > > > for > > > > > the coming 1.17 release. I'm all ears and looking forward to your > > > > feedback. > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > [3] https://github.com/apache/flink/pull/21172 > > > > > [4] https://github.com/apache/flink/pull/21171 > > > > > > > > > > > > > > >
