> they require additional hop to serialize Scala objects
This doesn't necessarily mean that we need a Scala API, because a beefed
up type extraction could also solve this.
> This single committer is now with us and ready to maintain it in open
source. The best situation to be :-)
Have you considered maintaining the wrappers as part of flink-extended?
https://github.com/flink-extended
On 17/04/2023 09:45, Alexey Novakov via user wrote:
Hi Günter, David,
Let me reply to you both in one email. First of all, thank you for
engaging.
Günter:
- I fully agree that losing Scala API as officially supported in Flink
would be very unfortunate. Future of Scala is interesting and will
bring more benefits to Flink users.
Just to remind everyone, Flink Scala users can't only use Java API,
they require additional hop to serialize Scala objects. This is one of
the reasons why Flink still has Scala API (2.11) and why a few more
3-rd party wrappers appeared to support newer versions of Scala when
it became possible.
David:
Let me address your concerns.
1. It is indeed not a very active project. This is exactly the reason,
I want to save https://github.com/findify/flink-scala-api from dying,
because it is quite a good library to work with. Our Idea is to hit
two targets: get a newer/official Scala API for Flink and do not let
the 3rd-party (currently) library to sink. I use this library for
daily work.
2. It works for Flink 1.15, support of Flink 1.16. requires just
publishing a new version. I guess it is a one line change in the
build.sbt file. Will see if more changes would be needed. I think the
nature of changes will be similar like in StateFun, i.e. adopt to
breaking changes of public methods and/or switch from deprecated
methods to newer alternatives. Migrating further should not be a
problem. Again, Scala API is supposed to be a thin wrapper on top of
Java API, so that it is not labour-intensive
3. That single person left Findify (Roman) and they did not pay much
attention to it. Actually, there is no other better alternative for
Scala wrapper currently. This single committer is now with us and
ready to maintain it in open source. The best situation to be :-)
4/5. Yes, same as #1. You can see some PRs in the queue from a Scala
bot, but Findify does not merge them. The library is so small and
covers most of the needs that additional changes are not yet
identified/needed. I agree this could be a sign that few people are
using it.
I have no idea which companies or users use this library. Is it
really important to know? I just want to provide proper substitution
to guarantee Scala is used further with Flink.
I know that the official Scala API was used or still used by world
known enterprises.
Thank you for your suggestion. I have included dev ML in the original
email. Let me try to come up with a more detailed plan.
Among maintainers you will get me, Roman (main dev of
https://github.com/findify/flink-scala-api) and maybe Günter.
What is the downside or loss if we import this library into the Flink
and in a few years nobody will use it? I guess we'll just depreciate it?
I just propose my free time to maintain that. As per Roman, required
work to maintain the library is very simple.
Best regards,
Alexey
On Sun, Apr 16, 2023 at 11:46 AM David Morávek <d...@apache.org> wrote:
cc dev@f.a.o
On Sun, Apr 16, 2023 at 11:42 AM David Morávek <d...@apache.org>
wrote:
Hi Alexey,
I'm a bit skeptical because, looking at the project, I see a
couple of red flags:
- The project is inactive. The last release and commit are
both from the last May.
- The project has not been adapted for the last two Flink
versions, which signals a lack of users.
- All commits are by a single person, which could mean that
there is no community around the project.
- There was no external contribution (except the Scala bot).
- There is no fork of the project (except the Scala bot).
> As I know, FIndify does not want or cannot maintain this
library.
Who are the users of the library? I'd assume Findify no longer
uses it if they're abandoning it.
> which would be similar to the StateFun
We're currently dealing with a lack of maintainers for
StateFun, so we should have a solid building ground around the
project to avoid the same issue.
I think there is value in having a modern Scala API, but we
should have a bigger plan to address the future of Flink Scala
APIs than importing an unmaintained library and calling it a
day. I suggest starting a thread on the dev ML and concluding
the overall plan first.
Best,
D.
On Sun, Apr 16, 2023 at 10:48 AM guenterh.lists
<guenterh.li...@bluewin.ch> wrote:
Hello Alexey
Thank you for your initiative and your suggestion!
I can only fully support the following statements in your
email:
>Taking into account my Scala experience for the last 8
years, I
predict these wrappers will eventually be abandoned,
unless such a Scala
library is a part of some bigger community like ASF.
>Also, non-official Scala API will lead people to play
safe and choose
Java API only, even if they didn't want that at the beginning.
Second sentence is my current state.
From my point of view it would be very unfortunate if the
Flink project
would lose the Scala API and thus the integration of
concise, flexible
and future-oriented language constructs of the Scala
language (and
further development of version 3).
Documentation of the API is essential. I would be
interested to support
this efforts.
Best wishes
Günter
On 13.04.23 15:39, Alexey Novakov via user wrote:
> Hello Flink PMCs and Flink Scala Users,
>
> I would like to propose an idea to take the 3rd party
Scala API
> findify/flink-scala-api
<https://github.com/findify/flink-scala-api>
> project into the Apache Flink organization.
>
> *Motivation *
>
> The Scala-free Flink idea was finally implemented by the
1.15 release and
> allowed Flink users to bring their own Scala version and
use it via the
> Flink Java API. See blog-post here: Scala Free in One
Fifteen
>
<https://flink.apache.org/2022/02/22/scala-free-in-one-fifteen/>.
Also,
> existing Flink Scala API will be deprecated, because it
is too hard to
> upgrade it to Scala 2.13 or 3.
>
> Taking into account my Scala experience for the last 8
years, I predict
> these wrappers will eventually be abandoned, unless such
a Scala library is
> a part of some bigger community like ASF.
> Also, non-official Scala API will lead people to play
safe and choose Java
> API only, even if they did want that at the beginning.
>
> https://github.com/findify/flink-scala-api has already
advanced and
> implemented Scala support for 2.13 and 3 versions on top
of Flink Java API.
> As I know, FIndify does not want or does not have a
capacity to maintain
> this library. I propose to fork this great library and
create a new Flink
> project with its own version and build process (SBT, not
Maven), which
> would be similar to the StateFun or FlinkML projects.
>
> *Proposal *
>
> 1. Create a fork of findify/flink-scala-api and host in
Apache Flink Git
> space (PMCs please advise).
> 2. I and Roman
>
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rgrebennikov>
> would
> be willing to maintain this library in future for the
next several years.
> Further, we believe it will live on its own.
> 3. Flink Docs: PMCs, we need your guidelines here. One
way I see is to
> create new documentation in a similar way as StateFun
docs. Alternatively,
> we could just fix existing Flink Scala code examples to
make sure they work
> with the new wrapper. In any case, I see docs will be
upgraded/fixed
> gradually.
>
> I hope you will find this idea interesting and worth
going forward.
>
> P.S. The irony here is that findify/flink-scala-api was
also a fork of
> Flink Scala-API some time ago, so we have a chance to
close the loop :-)
>
> Best regards.
> Alexey
>
--
Günter Hipler
https://openbiblio.social/@vog61
https://twitter.com/vog61
