As a yet another attempt to make Flink work with scala 2.13/3.x, we went further and and cross-built a forked version of Flink's Scala API: https://github.com/findify/flink-scala-api <https://github.com/findify/flink-scala-api.> Check the github repo for details, but if you can afford re-bootstrapping your job state from scratch (as there is no savepoint compatibility, see the repo readme for details), then migration is quite a straightforward process. Main features:
* Cross-built for Scala 2.12, 2.13 and 3.x * 100% replica of the 1.15 Scala API, with deprecated methods removed. * ClosureCleaner from Spark 3.x with 2.13 and java 17 support. * Agnostic to the serialization format, but can use flink-adt to support scala more natively (with sealed traits and scala3 typeclasses instead of scary macro). It's still an experimental version which is not yet well tested, but I don't see any possible dangers in it: * we're using flink-adt exclusively for a lot of Flink jobs in Findify for almost a year with no issues so far. * Flink's Scala API is a thin wrapper on top of Java API, so there is nothing to break. The only not well-tested thing is scala3 derivation, which is a compile-time process. So if you have compilation errors due to flink-adt failing for your case classes, you're welcome to submit a bug report on github. with best regards, Roman Grebennikov | [email protected] On Wed, May 11, 2022, at 07:38, Ran Tao wrote: > Hi, guys. I posted a jdk11 & jdk17 issue [FLINK-27549] > <https://issues.apache.org/jira/browse/FLINK-27549> recently which involved > upgrading scala [2] of current discussion. > > It shows that the current flink project is not a completed or pure jdk11 > version.(same problem with higher version). because the generated scala > bytecode <=1.8 can't generate target-11 or higher bytecode version. If scala > upgrade to 2.13 will fix this problem. > > Hope it helps~ > > [1] https://lists.apache.org/[email protected]:lte=1M:Ran%20Tao > > Martijn Visser <[email protected]> 于2022年5月11日周三 15:10写道： >> Hi Matthias, >> >> Given the current state of Scala support in the Flink community (there is a >> major lack in Scala maintainers), it is my personal opinion that we should >> consider deprecating the current Scala APIs and replace those with new Scala >> APIs, which are 'just' wrappers for the Java API. This definitely needs a >> FLIP and a discussion, where I can explain the situation more. There's >> nothing concrete for this yet though. There have been some discussions on >> this topic going on in the ticket for adding Scala 2.13 support [1] >> >> Best regards, >> >> Martijn >> >> [1] >> https://issues.apache.org/jira/browse/FLINK-13414?focusedCommentId=17344555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17344555 >> >> On Tue, 10 May 2022 at 16:16, Schwalbe Matthias >> <[email protected]> wrote: >>> … just for my understanding____ >>> __ __ >>> From the announcements I only got that scala remains only a dependency in >>> the JARs that relate to the Scala API.____ >>> I never read about plans to drop the Scala API altogether … is that the >>> case??____ >>> That would be very unfortunate …____ >>> __ __ >>> What is the state of the affair?____ >>> __ __ >>> Best regards____ >>> __ __ >>> Thias____ >>> __ __ >>> __ __ >>> __ __ >>> *From:* Martijn Visser <[email protected]> >>> *Sent:* Monday, May 9, 2022 2:38 PM >>> *To:* Robert Metzger <[email protected]> >>> *Cc:* Salva Alcántara <[email protected]>; user >>> <[email protected]> >>> *Subject:* Re: Practical guidance with Scala and Flink >= 1.15____ >>> __ __ >>> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠____ >>> __ __ >>> Hi Salva,____ >>> __ __ >>> Like Robert said, I don't expect that we will be able to drop support for >>> Scala 2.12 anytime soon. I do think that we should have a discussion in the >>> Flink community about providing Scala APIs. My opinion is that we are >>> probably better off to deprecate the current Scala APIs (keeping it >>> internal as we still have a big piece of Scala internally) and only offer >>> Java APIs. The Flink community lacks real Scala maintainers. I think Seth's >>> blog is pretty spot-on on this too [1].____ >>> __ __ >>> Best regards,____ >>> __ __ >>> Martijn Visser____ >>> https://twitter.com/MartijnVisser82____ >>> https://github.com/MartijnVisser____ >>> __ __ >>> [1] https://flink.apache.org/2022/02/22/scala-free.html____ >>> __ __ >>> On Mon, 9 May 2022 at 12:24, Robert Metzger <[email protected]> wrote:____ >>>> Hi Salva,____ >>>> my somewhat wild guess (because I'm not very involved with the Scala >>>> development on Flink): I would stick with option 1 for now. It should be >>>> easier now for the Flink community to support Scala versions past 2.12 >>>> (because we don't need to worry about scala 2.12+ support for Flink's >>>> internal dependencies such as akka).____ >>>> An argument against supporting newer Scala versions is that I'm not aware >>>> of anybody currently working on Flink with Scala in general.____ >>>> __ __ >>>> On Fri, May 6, 2022 at 6:37 PM Salva Alcántara <[email protected]> >>>> wrote:____ >>>>> I've always used Scala in the context of Flink. Now that Flink 1.15 has >>>>> become Scala-free, I wonder what is the best (most practical) route for >>>>> me moving forward. These are my options:____ >>>>> __ __ >>>>> 1. Keep using Scala 2.12 for the years to come (and upgrade to newer >>>>> versions when the community has come up with something). How long is >>>>> Flink expected to support Scala 2.12?____ >>>>> __ __ >>>>> 2. Upgrade to Scala 2.13 or Scala 3 and use the Java API directly >>>>> (without any Scala-specific wrapper/API). How problematic will that be, >>>>> especially regarding type information & scala-specific serializers? I >>>>> hate those "returns" (type hints) in the Java API...____ >>>>> __ __ >>>>> 3. Switch to Java, at least for the time being...____ >>>>> __ __ >>>>> To be clear, I have a strong preference for Scala over Java, but I'm >>>>> trying to look at the "grand scheme of things" here, and be pragmatic. I >>>>> guess I'm not alone here, and that many people are indeed evaluating the >>>>> same pros & cons. Any feedback will be much appreciated.____ >>>>> __ __ >>>>> Thanks in advance!____ >>> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und >>> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die >>> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, >>> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und >>> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir >>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie >>> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung >>> dieser Informationen ist streng verboten. >>> >>> This message is intended only for the named recipient and may contain >>> confidential or privileged information. As the confidentiality of email >>> communication cannot be guaranteed, we do not accept any responsibility for >>> the confidentiality and the intactness of this message. If you have >>> received it in error, please advise the sender by return e-mail and delete >>> this message and any attachments. Any unauthorised use or dissemination of >>> this information is strictly prohibited. > > > -- > Best, > Ran Tao
