Hi Chesnay
"Move Calcite rules from Scala to Java": I would hope that this would be
an entirely internal change, and could thus be an incremental process
independent of major releases.
What is the actual scale of this item; how much are we actually
re-writing?
Thanks for asking
yes, you're right, that should be internal change.
Yeah I was also thinking about incremental change (rule by rule or
reasonable small group of rules).
And yes, this could be an independent (on major release) activity
The problem is actually for children of RelOptRule.
Currently I see 60+ such rules (in Scala) using the mentioned deprecated
api.
There are also children of ConverterRule (50+) which do not have such
issues.
Maybe it could be considered as the next step to have all the rules in
Java.
On Tue, Jun 27, 2023 at 1:34 PM Xintong Song <tonysong...@gmail.com>
wrote:
Hi Alex & Gyula,
By compatibility discussion do you mean the "[DISCUSS] FLIP-321:
Introduce
an API deprecation process" thread [1]?
Yes, I meant the FLIP-321 discussion. I just noticed I pasted the wrong
url
in my previous email. Sorry for the mistake.
I am also curious to know if the rationale behind this new API has been
previously discussed on the mailing list. Do we have a list of
shortcomings
in the current DataStream API that it tries to resolve? How does the
current ProcessFunction functionality fit into the picture? Will it be
kept
as is or subsumed by new API?
I don't think we should create a replacement for the DataStream API
unless
we have a very good reason to do so and with a proper discussion about
this
as Alex said.
The ProcessFunction API which is targeting to replace DataStream API is
still a proposal, not a decision. Sorry for the confusion, I should have
been more careful with my words, not giving the impression that this is
something we'll do anyway.
There will be a FLIP describing the motivations and designs in detail,
for
the community to discuss and vote on. We are still working on it. TBH,
this
is not trivial and we would need more time on it.
Just to quickly share some backgrounds:
- We see quite some problems with the current DataStream APIs
- Users are working with concrete classes rather than interfaces,
which means
- Users can access methods that are designed to be used by internal
classes, even though they are annotated with `@Internal`. E.g.,
`DataStream#getTransformation`.
- Changes to the non-API implementations (e.g.,
`Transformation`)
would affect the API classes (e.g., `DataStream`), which
makes it hard to
provide binary compatibility.
- Internal classes are used as parameter / return-value of public
APIs. E.g., while `AbstractStreamOperator` is PublicEvolving,
`StreamTask`
which returns from `AbstractStreamOperator#getContainingTask` is
Internal.
- In many cases, users are asked to extend the API classes, rather
than implementing interfaces. E.g., `AbstractStreamOperator`.
- Any changes to the base classes, even the internal part, may
affect the behavior of the user-provided sub-classes
- Users can override the behavior of the base classes
- The API module `flink-streaming-java` contains non-API classes,
and
depends on internal modules such as `flink-runtime`, which means
- Changes to the internal modules may affect the API modules, which
requires users to re-build their applications upon upgrading
- The artifact user needs for building their application larger
than necessary.
- We probably should not expose operators (e.g.,
`AbstractStreamOperator`) to users. Functions should be enough
for users to
define their data processing logics. Exposing operator-level
concepts
(e.g., mailbox thread model, checkpoint barrier alignment, etc.) is
unnecessary and limits the improvement regarding such exposed
mechanisms
with compatibility considerations.
- The current DataStream API seems to be a mixture of many things,
making it hard to understand especially for newcomers. It might be
better
to re-organize it into several parts: (the taxonomy below are just
an
example of the, we are still working on this)
- The most fundamental stateful stream processing: streams,
partitions / key, process functions, state, timeline-service
- An extension for common batch-streaming unified functions:
map,
flatmap, filter, agg, reduce, join, etc.
- An extension for windowing supports: window, triggering
- An extension for event-time supports: event time, watermark
- The extensions are like short-cuts / sugars, without which
users
can probably still achieve the same behavior by working with the
fundamental APIs, but would be a lot easier with the extensions
- The original plan was to do in-place refactors / changes on
DataStream API. Some related items are listed in this doc [2] attached
to
the kicking off email [3]. Not all of the above issues are listed,
because
we haven't looked into this as deeply as now by that time.
- We proposed this as a new API rather than in-place refactors in the
2.0 work item list, because we realized the changes might be too big
for an
in-place change. First having a new API then gradually retiring the
old
one
would help users to smoothly migrate between them.
A thorough discussion is definitely needed once the FLIP is out. And of
course it's possible that the FLIP might be rejected. Given that we are
planning for release 2.0, I just feel it would be better to bring this up
early even the concrete plan is not yet ready,
Best,
Xintong
[1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9
[2]
https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing
[3] https://lists.apache.org/thread/b8w5cx0qqbwzzklyn5xxf54vw9ymys1c
On Tue, Jun 27, 2023 at 5:15 PM Gyula Fóra <gyf...@apache.org> wrote:
Hey!
I share the same concerns mentioned above regarding the
"ProcessFunction
API".
I don't think we should create a replacement for the DataStream API
unless
we have a very good reason to do so and with a proper discussion about
this
as Alex said.
Cheers,
Gyula
On Tue, Jun 27, 2023 at 11:03 AM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:
Hi Xintong,
By compatibility discussion do you mean the "[DISCUSS] FLIP-321:
Introduce
an API deprecation process" thread [1]?
I am also curious to know if the rationale behind this new API has
been
previously discussed on the mailing list. Do we have a list of
shortcomings
in the current DataStream API that it tries to resolve? How does the
current ProcessFunction functionality fit into the picture? Will it
be
kept
as is or subsumed by new API?
[1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9
Best,
Alex
On Mon, 26 Jun 2023 at 14:33, Xintong Song <tonysong...@gmail.com>
wrote:
The ProcessFunction API item is giving me the most headaches
because
it's
very unclear what it actually entails; like is it an entirely
separate
API
to DataStream (sounds like it is!) or an extension of DataStream.
How
much
will it share the internals with DataStream etc.; how does it
relate
to
the
Table API (w.r.t. switching APIs / what Table API uses
underneath).
I totally understand your confusion. We started planning this after
kicking
off the release 2.0, so there's still a lot to be explored and the
plan
keeps changing.
- In the beginning, we planned to do an in-place refactor of
DataStream
API, until the API migration period is proposed.
- Then we want to make it an entirely separate API to
DataStream,
and
listed as a must-have for release 2.0 so that we can remove
DataStream
once
it's ready.
- However, depending on the outcome of the API compatibility
discussion
[1], we may not be able to remove DataStream in 2.0 anyway,
which
means
we
might need to re-evaluate the necessity of this item for 2.0.
I'd say we wait a bit longer for the compatibility discussion [1]
and
decide the priority for this item afterwards.
Best,
Xintong
[1] https://lists.apache.org/list.html?dev@flink.apache.org
On Mon, Jun 26, 2023 at 6:00 PM Chesnay Schepler <
ches...@apache.org
wrote:
by-and-large I'm quite happy with the list of items.
I'm curious as to why the "Disaggregated State Management" item
is
marked
as a must-have; will it require changes that break something?
What
prevents
it from being added in 2.1?
We may want to update the Java 17 item to "Make Java 17 the
default,
drop
Java 8/11". Maybe even split it into a must-have "Drop Java 8"
and
a
nice-to-have "Drop Java 11"?
"Move Calcite rules from Scala to Java": I would hope that this
would
be
an entirely internal change, and could thus be an incremental
process
independent of major releases.
What is the actual scale of this item; how much are we actually
re-writing?
"Add MetricGroup#getLogicalScope": I'd raise this to a
must-have; i
think
I marked it down as nice-to-have only because it depends on
another
item.
The ProcessFunction API item is giving me the most headaches
because
it's
very unclear what it actually entails; like is it an entirely
separate
API
to DataStream (sounds like it is!) or an extension of DataStream.
How
much
will it share the internals with DataStream etc.; how does it
relate
to
the
Table API (w.r.t. switching APIs / what Table API uses
underneath).
There are a few items I added as ideas which don't have a
priority
yet;
would love to get some feedback on those.
On 21/06/2023 08:41, Xintong Song wrote:
Hi devs,
As previously discussed in [1], we had been collecting work item
proposals
for the 2.0 release until June 15th, on the wiki page [2].
- As we have passed the due date, I'd like to kindly remind
everyone
*not
to add / remove items directly on the wiki page*. If needed,
please
post
in this thread or reach out to the release managers instead.
- I've reached out to some folks for clarifications about
their
proposals. Some of them mentioned that they can not yet tell
whether
we
should do an item or not, and would need more time /
discussions
to
make
the decision. So I added a new symbol for items whose
priorities
are
`TBD`.
Now it's time to collaboratively decide a minimum set of
must-have
items.
I've gone through the entire list of proposed items, and found
most
of
them
make quite much sense. So I think an online sync might not be
necessary
for
this. I'd like to go with this DISCUSS thread, where everyone can
comment
on how they think the list can be improved, followed by a VOTE to
formally
make the decision.
Any feedback and opinions, including but not limited to the
following
aspects, will be appreciated.
- Important items that are missing from the list
- Concerns regarding the listed items or their priorities
Looking forward to your feedback.
Best,
Xintong
[1]
https://lists.apache.org/list?dev@flink.apache.org:lte=1M:release%202.0%20status%20updates
[2]
https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
--
Best regards,
Sergey
