Hi all, I just sent out the first release candidate for 3.4.0. The only difference to 3.3.0 is that we cease to support 1.19 in favor of supporting the open lineage interfaces [1] that have been added to Flink 1.20.
With that release under way (please verify and vote!), the main branch is now good to go for the Flink 2.0-preview release. @Qingsheng Ren <renqs...@gmail.com> and Leonard are driving that release, Yanquan already volunteered to help. I'm pretty sure more help is always welcome. Maybe it even makes sense to organize in Slack for that release. Best of luck and don't hesitate to reach out to me if you need another opinion. Arvid [1] https://issues.apache.org/jira/browse/FLINK-34466 On Wed, Nov 6, 2024 at 2:16 PM Leonard Xu <xbjt...@gmail.com> wrote: > Thanks @Yanquan for track this thread, and thanks @Arvid for the > information, it makes sense to me > > Qingsheng will drive the release of flink-2.0-preview of Flink Kafka > connector, and I’d like assist it too. > > Best, > Leonard > > > > 2024年11月6日 下午6:58，Arvid Heise <ahe...@confluent.io.INVALID> 写道： > > > > Hi Yanquan, > > > > the current state of the 3.4.0 release is that it's still pending on the > > lineage PR [1] which I expect to be merged next week (the author is on > > vacation). The release cut would then happen right afterwards. > > > > After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink > > 2.0-preview. @Qingsheng Ren <renqs...@gmail.com> and Leonard wanted to > > drive that release. I already prepared a bit by thoroughly annotating > > everything with Deprecated but the whole test side needs a bigger > cleanup. > > It's probably also a good time to bump other dependencies. > > Could you please sync with the two release managers? At least Qingsheng > is > > responsive in the Flink slack - I talked to him quite a bit there. > > > > If there is a pressing need to start earlier, we can also cut the 3.4 > > branch (which is then effectively the 3.3 branch) earlier and backport > the > > lineage PR (it's just one commit ultimately). I'd leave that decision to > > the two release managers for 4.0.0 mentioned before. > > > > One thing to note for 4.0.0 is that we need to solve the transaction > > management issues with the KafkaSink [2]. It's blocking larger users from > > adopting the KafkaSink which will be the only option for Flink 2.0. I > have > > started designing a solution. > > > > Best, > > > > Arvid > > > > > > [1] https://github.com/apache/flink-connector-kafka/pull/130 > > [2] https://issues.apache.org/jira/browse/FLINK-34554 > > > > On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <decq12y...@gmail.com> wrote: > > > >> Hi, Arvid. > >> > >> It has been a month and we are glad to see that we have completed the > >> release of Kafka 3.3.0 targeting 1.19 and 1.20. > >> > >> Considering that Flink 2.0-preview1 has already been released, I would > >> like to know about our plans and progress for bumping to 2.0-preview1. > >> I tested the changes required for bump to 2.0-preview1 locally and found > >> that the adaptation changes made in the production code based on > >> FlinkKafkaProducer Depreciated work were relatively clear and the > amount of > >> change was not significant. However, the headache was that there were > many > >> adjustments needed in the test code. > >> > >> I would like to know if there is already work in the community to bump > to > >> 2.0-preview1. If not, I can help complete this task (but some > suggestions > >> may be needed for testing the adaptation in the code). > >> > >> > >> > >> > >> > >>> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道： > >>> > >>> Dear Flink devs, > >>> > >>> I'd like to initiate three(!) Kafka connector releases. The main reason > >> for > >>> having three releases is that we have been slacking a bit in keeping up > >>> with the latest changes. > >>> > >>> Here is the summary: > >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap) > >>> - Incorporates lots of deprecations for Flink 2 including everything > that > >>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer > >>> (SourceFunction), and KafkaShuffle > >>> - Lots of bugfixes that are very relevant for 1.19 users (and probably > >> work > >>> with older releases) > >>> > >>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later) > >>> - Incorporates lineage tracing which is only available in 1.20 [1] > (FLIP > >>> incorrectly says that it's avail in 1.19) > >>> - We have discussed some alternatives to this release in [2] but > >> basically > >>> having a separate release is the cleanest solution. > >>> - I'd like to linearize the releases to avoid having to do back or even > >>> forward ports > >>> > >>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later) > >>> - Much requested to get the connector out asap for the preview. (I > think > >>> the old jar using the removed interfaces should still work) > >>> - Remove all deprecated things > >>> - General spring cleaning (trying to get rid of arch unit violations, > >>> migrate to JUnit5) > >>> - Should we relocate the TableAPI stuff to o.a.f.connectors? > >>> > >>> I'd appreciate any feedback and volunteers for RM ;) If you have > pending > >>> PRs that should be part of any of those releases, please also write > them. > >>> > >>> Best, > >>> > >>> Arvid > >>> > >>> [1] > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener > >>> [1] > >>> > >> > https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878 > >> > >> > >
