Thanks Martijn and David, And what about the FlinkKafkaProducer? In order to migrate it, the user has to make sure that the transactions stored on the state are committed, and that there are no lingering/pending transactions (lingering transactions are blocking further reads to be readable until they timeout). The first one can be achieved by stop-with-savepoint. Regular savepoint or checkpoint is not good enough, because transactions are committed on notfiyCheckpointCompleted RPC, not on the checkpoint itself (stop-with-savepoint is considered completed only once notifications are received). But what about the latter? What about the pending/lingering transactions? Are they also closed in stop-with-savepoint?
Best, Piotrek wt., 1 lis 2022 o 08:51 Martijn Visser <martijnvis...@apache.org> napisał(a): > Hi all, > > @Piotr > When FlinkKafkaConsumer was marked as deprecated, the migration steps were > included in the release notes: > > https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.14/#flink-24055httpsissuesapacheorgjirabrowseflink-24055 > Scrolling up a bit shows how move from FlinkKafkaProducer to KafkaSink > > Given the importance of the Kafka connector, I think it's smart to write a > dedicated blog on how to migrate. > That could also include the differences between the two implementations, as > David just outlined. > > @Jing > SinkV2 was released with Flink 1.15, so with 1.16 released there will be 2 > release cycles with that API at PublicEvolving. I don't think there were > breaking changes when the bug fixes were applied, because those bug fixes > were also applied in a patch version (1.15.3) which can't be done unless > they weren't breaking the API. Unless there will be work planned for 1.17 > on the Sink, I think it can be moved to Public too. > > Best regards, > > Martijn > > On Tue, Nov 1, 2022 at 7:21 AM David Anderson <dander...@apache.org> > wrote: > > > > > > > [H]ow one can migrate from the > > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while > > > preserving exactly-once guarantees etc? > > > > > > The responses from Fabian Paul in [1] and [2] address the question of how > > to handle the migration in terms of managing the state (where the short > > answer is "arrange for Kafka to be the source of truth"). > > > > Those threads don't get into the differences in behavior between the two > > implementations. Here I'm thinking about (1) the loss of > > DeserializationSchema#isEndOfStream, and the fact that you can no longer > > dynamically determine when the input stream has finished, and (2) the > > change to how empty partitions are handled on startup (they used to be > > marked idle automatically, whereas now you must use withIdleness in the > > WatermarkStrategy). > > > > [1] https://www.mail-archive.com/user@flink.apache.org/msg44618.html > > [2] https://www.mail-archive.com/user@flink.apache.org/msg45864.html > > > > On Mon, Oct 31, 2022 at 7:32 PM Piotr Nowojski <pnowoj...@apache.org> > > wrote: > > > > > Hi, > > > > > > Maybe a stupid question, but how one can migrate from the > > > FlinkKafkaConsumer/FlinkKafkaProducer to KafkaSource/KafkaSink, while > > > preserving exactly-once guarantees etc? Is it possible? I've tried a > > quick > > > search and couldn't find it, but maybe I was looking in wrong places. > > > > > > Best, > > > Piotrek > > > > > > pon., 31 paź 2022 o 16:40 Jing Ge <j...@ververica.com> napisał(a): > > > > > > > Thanks Martijn. What you said makes a lot of sense. I figure we > should > > do > > > > it in 2 steps. > > > > > > > > Step 1 (with 1.17): > > > > - Remove FlinkKafkaConsumer. > > > > - Graduate Kafka Source from @PublicEvolving to @Public. > > > > - Update doc and leave hints for customers as the reference. > > > > > > > > According to [1], the Kafka Sink should also be graduated with 1.17, > > i.e. > > > > after 1.15 and 1.16 two release cycles. But since the design change > > from > > > > SinkV1 to SinkV2 were significant and there were many change requests > > > since > > > > then, we'd better give the sink one more release cycle time to become > > > more > > > > stable. The other reason for giving the Sink more time is that the > > > > experimental phase was only covered by one release cycle instead of > two > > > as > > > > [1] suggested. > > > > > > > > Step 2 (with 1.18 ): > > > > - Remove FlinkKafkaProducer. > > > > - Graduate Kafka Sink from @PublicEvolving to @Public. > > > > - Update doc and leave hints for customers as the reference. > > > > > > > > Best regards, > > > > Jing > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > > > > > > On Thu, Oct 27, 2022 at 3:01 PM Martijn Visser < > > martijnvis...@apache.org > > > > > > > > wrote: > > > > > > > > > Hi Jing, > > > > > > > > > > Thanks for opening the discussion. I see no issue with removing the > > > > > FlinkKafkaConsumer, since it has been marked as deprecated and the > > > Source > > > > > API (which is used by the KafkaSource) is marked as @Public (at > least > > > the > > > > > Base implementation) > > > > > > > > > > The successor of the FlinkKafkaProducer is the KafkaSink, which is > > > using > > > > > the Sink V2 API which is still marked as @PublicEvolving (Base > > > > > implementation). I think that we should only remove the > > > > FlinkKafkaProducer > > > > > if we also mark the Sink V2 as @Public. I don't think that should > be > > a > > > > > problem (since it's based on the first Sink implementation, which > was > > > > > Experimental in 1.14 and got replaced with Sink V2 as > PublicEvolving > > in > > > > > 1.15). > > > > > > > > > > Thanks, > > > > > > > > > > Martijn > > > > > > > > > > On Thu, Oct 27, 2022 at 2:06 PM Jing Ge <j...@ververica.com> > wrote: > > > > > > > > > > > Hi Dev, > > > > > > > > > > > > I'd like to start a discussion about removing FlinkKafkaConsumer > > and > > > > > > FlinkKafkaProducer in 1.17. > > > > > > > > > > > > Back in the past, it was originally announced to remove it with > > Flink > > > > > 1.15 > > > > > > after Flink 1.14 had been released[1]. And then postponed to the > > next > > > > > 1.15 > > > > > > release which meant to remove it with Flink 1.16 but forgot to > > change > > > > the > > > > > > doc[2]. I have created a PRs to fix it. Since the 1.16 release > > branch > > > > has > > > > > > code freeze, it makes sense to, first of all, update the doc to > say > > > > that > > > > > > FlinkKafkaConsumer will be removed with Flink 1.17 [3][4] and > > second > > > > > start > > > > > > the discussion about removing them with the current master branch > > > i.e. > > > > > for > > > > > > the coming 1.17 release. I'm all ears and looking forward to your > > > > > feedback. > > > > > > Thanks! > > > > > > > > > > > > Best regards, > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > > [3] https://github.com/apache/flink/pull/21172 > > > > > > [4] https://github.com/apache/flink/pull/21171 > > > > > > > > > > > > > > > > > > > > >
