Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Wed, 05 Oct 2022 03:53:29 -0700

> It's possible that for the sake of the Scala API, we would occasionally require some changes in the Java API. As long as those changes are not detrimental to Java users, they should be considered.
That's exactly the model we're trying to get to. Don't fix scala-specific issues with scala code, but rather on the Java side as much as possible which could also benefit other JVM languages (e.g., Kotlin). 


> A question regarding the Flink wrapper: would it be possible to keep 
it under the Flink project's umbrella? Or does it need to be a 
completely separate structure? I'm not aware of the full organizational 
implications of this, I'm afraid.



Technically it can be under the Flink umbrella, but then Flink would 
still be (at least for a while) be the bottleneck because we'd have to 
review any changes coming in.
That would only improve once several new committers were added to take 
care of this project.
(In the end you'd just split Flink and the Scala API _codebases_, but 
achieve little else)



> And if that is what it takes to move beyond Scala 2.12.7… This has 
been a big pain point for us.


I'm curious what target Scala versions people are currently interested in.

I would've expected that everyone wants to migrate to Scala 3, for which 
several wrapper projects around Flink already exist.


On 05/10/2022 12:35, Gaël Renoux wrote:

Hello everyone,


I've already answered a bit on Twitter, I'll develop my thoughts a bit 
here. For context, my company (DataDome) has a significant codebase on 
Scala Flink (around 110K LOC), having been using it since 2017. I 
myself am an enthusiastic Scala developer (I don't think I'd like 
moving back to Java)



Given that, I think separating the Scala support from Flink is 
actually a good move long term. We would then have a full-Java Flink, 
and a separate Scala wrapper. It would help a lot in solving the 
skills issue: Flink maintainers would no longer need to be fluent in 
Scala, and maintainers of the Scala wrapper would not need a deep 
knowledge of Flink's inner workings, just the API would be sufficient. 
And if that is what it takes to move beyond Scala 2.12.7… This has 
been a big pain point for us.



I'm not too worried about finding contributors for the Scala wrapper. 
Within my company, we have developed additional wrappers and extension 
methods (for parts where we felt the Flink Scala API was 
insufficient), and we've been looking at ways we could contribute 
back. What held us back was our lack of knowledge of the full Flink 
environment (we're only using the Scala Datastream API). I don't think 
we're the only ones in that situation. One major point, though, is 
that Flink developers would not be completely rid of us ;-) It's 
possible that for the sake of the Scala API, we would occasionally 
require some changes in the Java API. As long as those changes are not 
detrimental to Java users, they should be considered.



A question regarding the Flink wrapper: would it be possible to keep 
it under the Flink project's umbrella? Or does it need to be a 
completely separate structure? I'm not aware of the full 
organizational implications of this, I'm afraid.



Finally, the hard part would be the migration to the new version. My 
dream solution would be to have the existing Scala API be entirely 
converted into a Scala wrapper over the Java API. That way, migration 
would be pretty minimal: add a dependency, change the imports for the 
Scala API, and we're done. However, even starting from the existing 
flink4s project, that's still quite a lot of work. So, more 
realistically, I'd settle for at least a partial implementation. We 
would have some broken code that we could fix, but at the very least 
I'd like the basic DataStream functions (process, uid, name…) to be 
available.


Thanks for all the work that went into making Flink what it is!


Gaël Renoux - Lead R&D Engineer
E - gael.ren...@datadome.co
W - www.datadome.co <http://www.datadome.co>



On Wed, Oct 5, 2022 at 9:30 AM Maciek Próchniak <m...@touk.pl> wrote:

    Hi Martin,

    Could you please remind what was the conclusion of discussion on
    upgrading Scala to 2.12.15/16?
    https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t -
    I couldn't find any follow-up vote?

    If it's acceptable to break binary compatibility by such an
    upgrade, then upgrading to JDK17 before 2.0 will be doable?


    thanks,

    maciek


    On 04.10.2022 18:21, Martijn Visser wrote:

    Hi Yaroslav,

    Thanks for the feedback, that's much appreciated! Regarding Java
    17 as a prerequisite, we would have to break compatibility
    already since Scala 2.12.7 doesn't compile on Java 17 [1].

    Given that we can only remove Scala APIs with the next major
    Flink (2.0) version, would that still impact you a lot? I do
    imagine that if we get to a Flink 2.0 version there would be more
    breaking involved anyway. The biggest consequence of deprecating
    support for Scala in Flink 1.x would be that new APIs would only
    be available in Java, but since these don't exist yet there would
    be no refactoring involved. I can imagine that we might change
    something in an existing API, but that would have certain
    compatibility guarantees already (depending if it's
    Public/PublicEvolving/Experimental). If a change would happen
    there, I think it would be smaller refactoring.

    Best regards,

    Martijn

    [1] https://issues.apache.org/jira/browse/FLINK-25000

    On Tue, Oct 4, 2022 at 10:58 AM Yaroslav Tkachenko
    <yaros...@goldsky.com> wrote:

        Hi Martijn,

        As a Scala user, this change would affect me a lot and I'm
        not looking forward to rewriting my codebase, and it's not
        even a very large one :)

        I'd like to suggest supporting Java 17 as a prerequisite
        (https://issues.apache.org/jira/browse/FLINK-15736). Things
        like switch expressions and records could simplify the
        migration quite a bit. Would you consider adding it to the FLIP?

        On Tue, Oct 4, 2022 at 10:50 AM Jing Ge <j...@ververica.com>
        wrote:

            Hi Martijn,

            Thanks for bringing this up. It is generally a great
            idea, so +1.

            Since both scala extension projects mentioned in the FLIP
            are still very young and I don't think they will attract
            more scala developers as Flink could just because they
            are external projects. It will be a big issue for users
            who have to rewrite their large codebases. Those users
            should be aware of the effort from now on and would
            better not count on those scala extension projects and
            prepare their migration plan before Flink 2.0.

            Best regards,
            Jing


            On Tue, Oct 4, 2022 at 1:59 PM Martijn Visser
            <martijnvis...@apache.org> wrote:

                Hi Marton,

                You're making a good point, I originally wanted to
                include already the User mailing list to get their
                feedback but forgot to do so. I'll do some more
                outreach via other channels as well.

                @Users of Flink, I've made a proposal to deprecate
                and remove Scala API support in a future version of
                Flink. Your feedback on this topic is very much
                appreciated.

                Regarding the large Scala codebase for Flink, a
                potential alternative could be to have a wrapper for
                all Java APIs that makes them available as Scala
                APIs. However, this still requires Scala maintainers
                and I don't think that we currently have those in our
                community. The easiest solution for them would be to
                use the Java APIs directly. Yes it would involve
                work, but we won't actually be able to remove the
                Scala APIs until Flink 2.0 so there's still time for
                that :)

                Best regards,

                Martijn

                On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
                <balassi.mar...@gmail.com> wrote:

                    Hi Martjin,

                    Thanks for compiling the FLIP. I agree with the
                    sentiment that Scala poses
                    considerable maintenance overhead and key
                    improvements (like 2.13 or 2.12.8
                    supports) are hanging stale. With that said
                    before we make this move we
                    should attempt to understand the userbase affected.
                    A quick Slack and user mailing list search does
                    return quite a bit of
                    results for scala (admittedly a cursory look at
                    them suggest that many of
                    them have to do with missing features in Scala
                    that exist in Java or Scala
                    versions). I would love to see some polls on this
                    topic, we could also use
                    the Flink twitter handle to ask the community
                    about this.

                    I am aware of users having large existing Scala
                    codebases for Flink. This
                    move would pose a very large effort on them, as
                    they would need to rewrite
                    much of their existing code. What are the
                    alternatives in your opinion,
                    Martjin?

                    On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser
                    <martijnvis...@apache.org>
                    wrote:

                    > Hi everyone,
                    >
                    > I would like to open a discussion thread on
                    FLIP-265 Deprecate and remove
                    > Scala API support. Please take a look at
                    >
                    >
                    
https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
                    > and provide your feedback.
                    >
                    > Best regards,
                    >
                    > Martijn
                    > https://twitter.com/MartijnVisser82
                    > https://github.com/MartijnVisser
                    >























					Reply via email to