> I'm curious what target Scala versions people are currently interested in. > I would've expected that everyone wants to migrate to Scala 3, for which several wrapper projects around Flink already exist
The Scala 3 tooling is still subpar (we're using IntelliJ), so I'm not sure I would move my team to Scala 3 right now (I'm currently toying with it on a personal project). In addition, moving to Scala 3 is not completely free - you have to do some rewrites, and developers will need some adaptation time. Scala 2.13 is another thing entirely, we've wanted to migrate for a long while. On Wed, Oct 5, 2022 at 12:53 PM Chesnay Schepler <ches...@apache.org> wrote: > > It's possible that for the sake of the Scala API, we would occasionally > require some changes in the Java API. As long as those changes are not > detrimental to Java users, they should be considered. > > That's exactly the model we're trying to get to. Don't fix scala-specific > issues with scala code, but rather on the Java side as much as possible > which could also benefit other JVM languages (e.g., Kotlin). > > > A question regarding the Flink wrapper: would it be possible to keep it > under the Flink project's umbrella? Or does it need to be a completely > separate structure? I'm not aware of the full organizational implications > of this, I'm afraid. > > Technically it can be under the Flink umbrella, but then Flink would still > be (at least for a while) be the bottleneck because we'd have to review any > changes coming in. > That would only improve once several new committers were added to take > care of this project. > (In the end you'd just split Flink and the Scala API _codebases_, but > achieve little else) > > > And if that is what it takes to move beyond Scala 2.12.7… This has been > a big pain point for us. > > I'm curious what target Scala versions people are currently interested in. > I would've expected that everyone wants to migrate to Scala 3, for which > several wrapper projects around Flink already exist. > > On 05/10/2022 12:35, Gaël Renoux wrote: > > Hello everyone, > > I've already answered a bit on Twitter, I'll develop my thoughts a bit > here. For context, my company (DataDome) has a significant codebase on > Scala Flink (around 110K LOC), having been using it since 2017. I myself am > an enthusiastic Scala developer (I don't think I'd like moving back to > Java) > > Given that, I think separating the Scala support from Flink is actually a > good move long term. We would then have a full-Java Flink, and a separate > Scala wrapper. It would help a lot in solving the skills issue: Flink > maintainers would no longer need to be fluent in Scala, and maintainers of > the Scala wrapper would not need a deep knowledge of Flink's inner > workings, just the API would be sufficient. And if that is what it takes to > move beyond Scala 2.12.7… This has been a big pain point for us. > > I'm not too worried about finding contributors for the Scala wrapper. > Within my company, we have developed additional wrappers and extension > methods (for parts where we felt the Flink Scala API was insufficient), and > we've been looking at ways we could contribute back. What held us back was > our lack of knowledge of the full Flink environment (we're only using the > Scala Datastream API). I don't think we're the only ones in that situation. > One major point, though, is that Flink developers would not be completely > rid of us ;-) It's possible that for the sake of the Scala API, we would > occasionally require some changes in the Java API. As long as those changes > are not detrimental to Java users, they should be considered. > > A question regarding the Flink wrapper: would it be possible to keep it > under the Flink project's umbrella? Or does it need to be a completely > separate structure? I'm not aware of the full organizational implications > of this, I'm afraid. > > Finally, the hard part would be the migration to the new version. My dream > solution would be to have the existing Scala API be entirely converted into > a Scala wrapper over the Java API. That way, migration would be pretty > minimal: add a dependency, change the imports for the Scala API, and we're > done. However, even starting from the existing flink4s project, that's > still quite a lot of work. So, more realistically, I'd settle for at least > a partial implementation. We would have some broken code that we could fix, > but at the very least I'd like the basic DataStream functions (process, > uid, name…) to be available. > > Thanks for all the work that went into making Flink what it is! > > > Gaël Renoux - Lead R&D Engineer > E - gael.ren...@datadome.co > W - www.datadome.co > > > > On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <m...@touk.pl> wrote: > >> Hi Martin, >> >> Could you please remind what was the conclusion of discussion on >> upgrading Scala to 2.12.15/16? >> https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I >> couldn't find any follow-up vote? >> >> If it's acceptable to break binary compatibility by such an upgrade, then >> upgrading to JDK17 before 2.0 will be doable? >> >> >> thanks, >> >> maciek >> >> >> On 04.10.2022 18:21, Martijn Visser wrote: >> >> Hi Yaroslav, >> >> Thanks for the feedback, that's much appreciated! Regarding Java 17 as a >> prerequisite, we would have to break compatibility already since Scala >> 2.12.7 doesn't compile on Java 17 [1]. >> >> Given that we can only remove Scala APIs with the next major Flink (2.0) >> version, would that still impact you a lot? I do imagine that if we get to >> a Flink 2.0 version there would be more breaking involved anyway. The >> biggest consequence of deprecating support for Scala in Flink 1.x would be >> that new APIs would only be available in Java, but since these don't exist >> yet there would be no refactoring involved. I can imagine that we might >> change something in an existing API, but that would have certain >> compatibility guarantees already (depending if it's >> Public/PublicEvolving/Experimental). If a change would happen there, I >> think it would be smaller refactoring. >> >> Best regards, >> >> Martijn >> >> [1] https://issues.apache.org/jira/browse/FLINK-25000 >> >> On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko <yaros...@goldsky.com> >> wrote: >> >>> Hi Martijn, >>> >>> As a Scala user, this change would affect me a lot and I'm not looking >>> forward to rewriting my codebase, and it's not even a very large one :) >>> >>> I'd like to suggest supporting Java 17 as a prerequisite ( >>> https://issues.apache.org/jira/browse/FLINK-15736). Things like switch >>> expressions and records could simplify the migration quite a bit. Would you >>> consider adding it to the FLIP? >>> >>> On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <j...@ververica.com> wrote: >>> >>>> Hi Martijn, >>>> >>>> Thanks for bringing this up. It is generally a great idea, so +1. >>>> >>>> Since both scala extension projects mentioned in the FLIP are still >>>> very young and I don't think they will attract more scala developers as >>>> Flink could just because they are external projects. It will be a big issue >>>> for users who have to rewrite their large codebases. Those users should be >>>> aware of the effort from now on and would better not count on those scala >>>> extension projects and prepare their migration plan before Flink 2.0. >>>> >>>> Best regards, >>>> Jing >>>> >>>> >>>> On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser <martijnvis...@apache.org> >>>> wrote: >>>> >>>>> Hi Marton, >>>>> >>>>> You're making a good point, I originally wanted to include already the >>>>> User mailing list to get their feedback but forgot to do so. I'll do some >>>>> more outreach via other channels as well. >>>>> >>>>> @Users of Flink, I've made a proposal to deprecate and remove Scala >>>>> API support in a future version of Flink. Your feedback on this topic is >>>>> very much appreciated. >>>>> >>>>> Regarding the large Scala codebase for Flink, a potential alternative >>>>> could be to have a wrapper for all Java APIs that makes them available as >>>>> Scala APIs. However, this still requires Scala maintainers and I don't >>>>> think that we currently have those in our community. The easiest solution >>>>> for them would be to use the Java APIs directly. Yes it would involve >>>>> work, >>>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so >>>>> there's still time for that :) >>>>> >>>>> Best regards, >>>>> >>>>> Martijn >>>>> >>>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi < >>>>> balassi.mar...@gmail.com> wrote: >>>>> >>>>>> Hi Martjin, >>>>>> >>>>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala >>>>>> poses >>>>>> considerable maintenance overhead and key improvements (like 2.13 or >>>>>> 2.12.8 >>>>>> supports) are hanging stale. With that said before we make this move >>>>>> we >>>>>> should attempt to understand the userbase affected. >>>>>> A quick Slack and user mailing list search does return quite a bit of >>>>>> results for scala (admittedly a cursory look at them suggest that >>>>>> many of >>>>>> them have to do with missing features in Scala that exist in Java or >>>>>> Scala >>>>>> versions). I would love to see some polls on this topic, we could >>>>>> also use >>>>>> the Flink twitter handle to ask the community about this. >>>>>> >>>>>> I am aware of users having large existing Scala codebases for Flink. >>>>>> This >>>>>> move would pose a very large effort on them, as they would need to >>>>>> rewrite >>>>>> much of their existing code. What are the alternatives in your >>>>>> opinion, >>>>>> Martjin? >>>>>> >>>>>> On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser < >>>>>> martijnvis...@apache.org> >>>>>> wrote: >>>>>> >>>>>> > Hi everyone, >>>>>> > >>>>>> > I would like to open a discussion thread on FLIP-265 Deprecate and >>>>>> remove >>>>>> > Scala API support. Please take a look at >>>>>> > >>>>>> > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support >>>>>> > and provide your feedback. >>>>>> > >>>>>> > Best regards, >>>>>> > >>>>>> > Martijn >>>>>> > https://twitter.com/MartijnVisser82 >>>>>> > https://github.com/MartijnVisser >>>>>> > >>>>>> >>>>> >
