Hi, I've just realized that it seems like KafkaSink is missing a feature that was present in FlinkKafkaProducer and which makes it hard to use with savepoints [1].
Best, Piotrek [1] https://issues.apache.org/jira/browse/FLINK-30068 pt., 11 lis 2022 o 13:53 Jing Ge <j...@ververica.com.invalid> napisał(a): > Hi all, > > Thank you all for the informative feedback. I figure there is a requirement > to improve the documentation wrt the migration from FlinkKafkaConsumer to > KafkaSource. I've fired a ticket[1] and connected it with [2]. This > shouldn't be the blocker for removing FlinkKafkaConsumer. > > Given there will be some ongoing SinkV2 upgrades, I will start a vote only > limited to FlinkKafkaConsumer elimination and related APIs graduation. As a > follow-up task, I will sync with Yun Gao before the coding freeze of 1.17 > release to check if we can start the second vote to remove > FlinkKafkaProducer with 1.17. > > Best regards, > Jing > > [1] https://issues.apache.org/jira/browse/FLINK-29999 > [2] https://issues.apache.org/jira/browse/FLINK-28302 > > > On Wed, Nov 2, 2022 at 11:39 AM Martijn Visser <martijnvis...@apache.org> > wrote: > > > Hi David, > > > > I believe that for the DataStream this is indeed documented [1] but it > > might be missed given that there is a lot of documentation and you need > to > > know that your problem is related to idleness. For the Table API I think > > this is never mentioned, so it should definitely be at least documented > > there. > > > > Thanks, > > > > Martijn > > > > [1] > > > > > https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/#idleness > > > > On Wed, Nov 2, 2022 at 11:28 AM David Anderson <dander...@apache.org> > > wrote: > > > > > > > > > > For the partition > > > > idleness problem could you elaborate more about it? I assume both > > > > FlinkKafkaConsumer and KafkaSource need a WatermarkStrategy to decide > > > > whether to mark the partition as idle. > > > > > > > > > As a matter of fact, no, that's not the case -- which is why I > mentioned > > > it. > > > > > > The FlinkKafkaConsumer automatically treats all initially empty (or > > > non-existent) partitions as idle, while the KafkaSource only does this > if > > > the WatermarkStrategy specifies that idleness handling is desired by > > > configuring withIdleness. This can be a source of confusion for folks > > > upgrading to the new connector. It most often shows up in situations > > where > > > the number of Kafka partitions is less than the parallelism of the > > > connector, which is a rather common occurrence in development and > testing > > > environments. > > > > > > I believe this change in behavior was made deliberately, so as to > create > > a > > > more consistent experience across all FLIP-27 connectors. This isn't > > > something that needs to be fixed, but does need to be communicated more > > > clearly. Unfortunately, the whole idleness mechanism remained > > significantly > > > broken until 1.16 (considering the impact of [1] and [2]), further > > > complicating the situation. Because of FLINK-28975 [2], users with > > > partitions that are initially empty may have problems with versions > > before > > > 1.15.3 (still unreleased) and 1.16.0. See [3] for an example of this > > > confusion. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-18934 (idleness didn't > > > work > > > with connected streams) > > > [2] https://issues.apache.org/jira/browse/FLINK-28975 (idle streams > > could > > > never become active again) > > > [3] > > > > > > > > > https://stackoverflow.com/questions/70096166/parallelism-in-flink-kafka-source-causes-nothing-to-execute/70101290#70101290 > > > > > > Best, > > > David > > > > > > On Wed, Nov 2, 2022 at 5:26 AM Qingsheng Ren <re...@apache.org> wrote: > > > > > > > Thanks Jing for starting the discussion. > > > > > > > > +1 for removing FlinkKafkaConsumer, as KafkaSource has evolved for > many > > > > release cycles and should be stable enough. I have some concerns > about > > > the > > > > new Kafka sink based on sink v2, as sink v2 still has some ongoing > work > > > in > > > > 1.17 (maybe Yun Gao could provide some inputs). Also we found some > > issues > > > > of KafkaSink related to the internal mechanism of sink v2, like > > > > FLINK-29492. > > > > > > > > @David > > > > About the ability of DeserializationSchema#isEndOfStream, FLIP-208 is > > > > trying to complete this piece of the puzzle, and Hang Ruan ( > > > > ruanhang1...@gmail.com) plans to work on it in 1.17. For the > partition > > > > idleness problem could you elaborate more about it? I assume both > > > > FlinkKafkaConsumer and KafkaSource need a WatermarkStrategy to decide > > > > whether to mark the partition as idle. > > > > > > > > Best, > > > > Qingsheng > > > > Ververica (Alibaba) > > > > > > > > On Thu, Oct 27, 2022 at 8:06 PM Jing Ge <j...@ververica.com> wrote: > > > > > > > > > Hi Dev, > > > > > > > > > > I'd like to start a discussion about removing FlinkKafkaConsumer > and > > > > > FlinkKafkaProducer in 1.17. > > > > > > > > > > Back in the past, it was originally announced to remove it with > Flink > > > > 1.15 > > > > > after Flink 1.14 had been released[1]. And then postponed to the > next > > > > 1.15 > > > > > release which meant to remove it with Flink 1.16 but forgot to > change > > > the > > > > > doc[2]. I have created a PRs to fix it. Since the 1.16 release > branch > > > has > > > > > code freeze, it makes sense to, first of all, update the doc to say > > > that > > > > > FlinkKafkaConsumer will be removed with Flink 1.17 [3][4] and > second > > > > start > > > > > the discussion about removing them with the current master branch > > i.e. > > > > for > > > > > the coming 1.17 release. I'm all ears and looking forward to your > > > > feedback. > > > > > Thanks! > > > > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#kafka-sourcefunction > > > > > [3] https://github.com/apache/flink/pull/21172 > > > > > [4] https://github.com/apache/flink/pull/21171 > > > > > > > > > > > > > > >
