Thanks all for the discussion.
IIUC, we need to make the following changes. Please correct me if I get it wrong. 1. Disaggregated State Management - Clarify that only the public API related part is must-have for 2.0. 2. Java version support - Split it into 3 items: a) make java 17 the default (must-have), b) drop java 8 (must-have), and c) drop java 11 (nice-to-have) 3. Add MetricGroup#getLogicalScope - Should be promoted to must-have 4. ProcessFunction API - Should be downgrade to nice-to-have 5. Configuration - Add an item "revisit all config option types and default values", which IIUC should also be a must-have There seems to be no changes needed for "Move Calcite rules from Scala to Java" as it's already nice-to-have. If there's no objections, I'll update the wiki page accordingly, and start a VOTE in the next couple of days. Best, Xintong On Fri, Jun 30, 2023 at 12:53 AM Teoh, Hong <lian...@amazon.co.uk.invalid> wrote: > Thanks Xintong for driving the effort. > > I’d add a +1 to reworking configs, as suggested by @Jark and @Chesnay, > especially the types. We have various configs that encode Time / MemorySize > that are Long instead! > > Regards, > Hong > > > > > On 29 Jun 2023, at 16:19, Yuan Mei <yuanmei.w...@gmail.com> wrote: > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > Thanks for driving this effort, Xintong! > > > > To Chesnay > >> I'm curious as to why the "Disaggregated State Management" item is > >> marked as a must-have; will it require changes that break something? > >> What prevents it from being added in 2.1? > > > > As to "Disaggregated State Management". > > > > We plan to provide a new type of state backend to support DFS as primary > > storage. > > To achieve this, we at least need to include two parts of amends (not > > entirely sure yet, since we are still in the designing and prototype > phase) > > > > 1. Statebackend Change > > 2. State Access Change > > > > Not all of the interfaces related are `@Internal`. Some of the interfaces > > like `StateBackend` is `@PublicEvolving` > > So, you are right in the sense that "Disaggregated State Management" > itself > > probably does not need to be a "Must Have" > > > > But I was hoping changes that related to public APIs can be finalized and > > merged in Flink 2.0 (I will fix the wiki accordingly). > > > > I also agree with Jark that 2.0 is a good chance to rework the default > > value of configurations. > > > > Best > > Yuan > > > > > > On Thu, Jun 29, 2023 at 8:43 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > >> Something else configuration-related is that there are a bunch of > >> options where the type isn't quite correct (e.g., a String where it > >> could be an enum, a string where it should be an int or something). > >> Could do a pass over those as well. > >> > >> On 29/06/2023 13:50, Jark Wu wrote: > >>> Hi, > >>> > >>> I think one more thing we need to consider to do in 2.0 is changing the > >>> default value of configuration to improve out-of-box user experience. > >>> > >>> Currently, in order to run a Flink job, users may need to set > >>> a bunch of configurations, such as minibatch, checkpoint interval, > >>> exactly-once, > >>> incremental-checkpoint, etc. It's very verbose and hard to use for > >>> beginners. > >>> Most of them can have a universally applicable value. Because changing > >> the > >>> default value is a breaking change. I think It's worth considering > >> changing > >>> them in 2.0. > >>> > >>> What do you think? > >>> > >>> Best, > >>> Jark > >>> > >>> > >>> On Wed, 28 Jun 2023 at 14:10, Sergey Nuyanzin <snuyan...@gmail.com> > >> wrote: > >>> > >>>> Hi Chesnay > >>>> > >>>>> "Move Calcite rules from Scala to Java": I would hope that this would > >> be > >>>>> an entirely internal change, and could thus be an incremental process > >>>>> independent of major releases. > >>>>> What is the actual scale of this item; how much are we actually > >>>> re-writing? > >>>> > >>>> Thanks for asking > >>>> yes, you're right, that should be internal change. > >>>> Yeah I was also thinking about incremental change (rule by rule or > >>>> reasonable small group of rules). > >>>> And yes, this could be an independent (on major release) activity > >>>> > >>>> The problem is actually for children of RelOptRule. > >>>> Currently I see 60+ such rules (in Scala) using the mentioned > deprecated > >>>> api. > >>>> There are also children of ConverterRule (50+) which do not have such > >>>> issues. > >>>> Maybe it could be considered as the next step to have all the rules in > >>>> Java. > >>>> > >>>> On Tue, Jun 27, 2023 at 1:34 PM Xintong Song <tonysong...@gmail.com> > >>>> wrote: > >>>> > >>>>> Hi Alex & Gyula, > >>>>> > >>>>> By compatibility discussion do you mean the "[DISCUSS] FLIP-321: > >>>> Introduce > >>>>>> an API deprecation process" thread [1]? > >>>>>> > >>>>> Yes, I meant the FLIP-321 discussion. I just noticed I pasted the > wrong > >>>> url > >>>>> in my previous email. Sorry for the mistake. > >>>>> > >>>>> I am also curious to know if the rationale behind this new API has > been > >>>>>> previously discussed on the mailing list. Do we have a list of > >>>>> shortcomings > >>>>>> in the current DataStream API that it tries to resolve? How does the > >>>>>> current ProcessFunction functionality fit into the picture? Will it > be > >>>>> kept > >>>>>> as is or subsumed by new API? > >>>>>> > >>>>> I don't think we should create a replacement for the DataStream API > >>>> unless > >>>>>> we have a very good reason to do so and with a proper discussion > about > >>>>> this > >>>>>> as Alex said. > >>>>> > >>>>> The ProcessFunction API which is targeting to replace DataStream API > is > >>>>> still a proposal, not a decision. Sorry for the confusion, I should > >> have > >>>>> been more careful with my words, not giving the impression that this > is > >>>>> something we'll do anyway. > >>>>> > >>>>> There will be a FLIP describing the motivations and designs in > detail, > >>>> for > >>>>> the community to discuss and vote on. We are still working on it. > TBH, > >>>> this > >>>>> is not trivial and we would need more time on it. > >>>>> > >>>>> Just to quickly share some backgrounds: > >>>>> > >>>>> - We see quite some problems with the current DataStream APIs > >>>>> - Users are working with concrete classes rather than > >> interfaces, > >>>>> which means > >>>>> - Users can access methods that are designed to be used by > >> internal > >>>>> classes, even though they are annotated with `@Internal`. > >> E.g., > >>>>> `DataStream#getTransformation`. > >>>>> - Changes to the non-API implementations (e.g., > >>>> `Transformation`) > >>>>> would affect the API classes (e.g., `DataStream`), which > >>>>> makes it hard to > >>>>> provide binary compatibility. > >>>>> - Internal classes are used as parameter / return-value of > >> public > >>>>> APIs. E.g., while `AbstractStreamOperator` is PublicEvolving, > >>>>> `StreamTask` > >>>>> which returns from `AbstractStreamOperator#getContainingTask` > is > >>>>> Internal. > >>>>> - In many cases, users are asked to extend the API classes, > >> rather > >>>>> than implementing interfaces. E.g., `AbstractStreamOperator`. > >>>>> - Any changes to the base classes, even the internal part, > >> may > >>>>> affect the behavior of the user-provided sub-classes > >>>>> - Users can override the behavior of the base classes > >>>>> - The API module `flink-streaming-java` contains non-API > >> classes, > >>>> and > >>>>> depends on internal modules such as `flink-runtime`, which > means > >>>>> - Changes to the internal modules may affect the API modules, > >> which > >>>>> requires users to re-build their applications upon upgrading > >>>>> - The artifact user needs for building their application > >> larger > >>>>> than necessary. > >>>>> - We probably should not expose operators (e.g., > >>>>> `AbstractStreamOperator`) to users. Functions should be enough > >>>>> for users to > >>>>> define their data processing logics. Exposing operator-level > >>>> concepts > >>>>> (e.g., mailbox thread model, checkpoint barrier alignment, > >> etc.) is > >>>>> unnecessary and limits the improvement regarding such exposed > >>>>> mechanisms > >>>>> with compatibility considerations. > >>>>> - The current DataStream API seems to be a mixture of many > >> things, > >>>>> making it hard to understand especially for newcomers. It might > >> be > >>>>> better > >>>>> to re-organize it into several parts: (the taxonomy below are > >> just > >>>> an > >>>>> example of the, we are still working on this) > >>>>> - The most fundamental stateful stream processing: streams, > >>>>> partitions / key, process functions, state, timeline-service > >>>>> - An extension for common batch-streaming unified functions: > >>>> map, > >>>>> flatmap, filter, agg, reduce, join, etc. > >>>>> - An extension for windowing supports: window, triggering > >>>>> - An extension for event-time supports: event time, > watermark > >>>>> - The extensions are like short-cuts / sugars, without which > >>>> users > >>>>> can probably still achieve the same behavior by working with > >> the > >>>>> fundamental APIs, but would be a lot easier with the > >> extensions > >>>>> - The original plan was to do in-place refactors / changes on > >>>>> DataStream API. Some related items are listed in this doc [2] > >> attached > >>>>> to > >>>>> the kicking off email [3]. Not all of the above issues are listed, > >>>>> because > >>>>> we haven't looked into this as deeply as now by that time. > >>>>> - We proposed this as a new API rather than in-place refactors in > >> the > >>>>> 2.0 work item list, because we realized the changes might be too > >> big > >>>>> for an > >>>>> in-place change. First having a new API then gradually retiring > the > >>>> old > >>>>> one > >>>>> would help users to smoothly migrate between them. > >>>>> > >>>>> A thorough discussion is definitely needed once the FLIP is out. And > of > >>>>> course it's possible that the FLIP might be rejected. Given that we > are > >>>>> planning for release 2.0, I just feel it would be better to bring > this > >> up > >>>>> early even the concrete plan is not yet ready, > >>>>> > >>>>> Best, > >>>>> > >>>>> Xintong > >>>>> > >>>>> > >>>>> [1] https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > >>>>> [2] > >>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > >>>>> [3] https://lists.apache.org/thread/b8w5cx0qqbwzzklyn5xxf54vw9ymys1c > >>>>> > >>>>> On Tue, Jun 27, 2023 at 5:15 PM Gyula Fóra <gyf...@apache.org> > wrote: > >>>>> > >>>>>> Hey! > >>>>>> > >>>>>> I share the same concerns mentioned above regarding the > >>>> "ProcessFunction > >>>>>> API". > >>>>>> > >>>>>> I don't think we should create a replacement for the DataStream API > >>>>> unless > >>>>>> we have a very good reason to do so and with a proper discussion > about > >>>>> this > >>>>>> as Alex said. > >>>>>> > >>>>>> Cheers, > >>>>>> Gyula > >>>>>> > >>>>>> On Tue, Jun 27, 2023 at 11:03 AM Alexander Fedulov < > >>>>>> alexander.fedu...@gmail.com> wrote: > >>>>>> > >>>>>>> Hi Xintong, > >>>>>>> > >>>>>>> By compatibility discussion do you mean the "[DISCUSS] FLIP-321: > >>>>>> Introduce > >>>>>>> an API deprecation process" thread [1]? > >>>>>>> > >>>>>>> I am also curious to know if the rationale behind this new API has > >>>> been > >>>>>>> previously discussed on the mailing list. Do we have a list of > >>>>>> shortcomings > >>>>>>> in the current DataStream API that it tries to resolve? How does > the > >>>>>>> current ProcessFunction functionality fit into the picture? Will it > >>>> be > >>>>>> kept > >>>>>>> as is or subsumed by new API? > >>>>>>> > >>>>>>> [1] > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > >>>>>>> > >>>>>>> Best, > >>>>>>> Alex > >>>>>>> > >>>>>>> On Mon, 26 Jun 2023 at 14:33, Xintong Song <tonysong...@gmail.com> > >>>>>> wrote: > >>>>>>>>> The ProcessFunction API item is giving me the most headaches > >>>>> because > >>>>>>> it's > >>>>>>>>> very unclear what it actually entails; like is it an entirely > >>>>>> separate > >>>>>>>> API > >>>>>>>>> to DataStream (sounds like it is!) or an extension of DataStream. > >>>>> How > >>>>>>>> much > >>>>>>>>> will it share the internals with DataStream etc.; how does it > >>>>> relate > >>>>>> to > >>>>>>>> the > >>>>>>>>> Table API (w.r.t. switching APIs / what Table API uses > >>>> underneath). > >>>>>>>> I totally understand your confusion. We started planning this > after > >>>>>>> kicking > >>>>>>>> off the release 2.0, so there's still a lot to be explored and the > >>>>> plan > >>>>>>>> keeps changing. > >>>>>>>> > >>>>>>>> > >>>>>>>> - In the beginning, we planned to do an in-place refactor of > >>>>>>> DataStream > >>>>>>>> API, until the API migration period is proposed. > >>>>>>>> - Then we want to make it an entirely separate API to > >>>> DataStream, > >>>>>> and > >>>>>>>> listed as a must-have for release 2.0 so that we can remove > >>>>>> DataStream > >>>>>>>> once > >>>>>>>> it's ready. > >>>>>>>> - However, depending on the outcome of the API compatibility > >>>>>>> discussion > >>>>>>>> [1], we may not be able to remove DataStream in 2.0 anyway, > >>>> which > >>>>>>> means > >>>>>>>> we > >>>>>>>> might need to re-evaluate the necessity of this item for 2.0. > >>>>>>>> > >>>>>>>> I'd say we wait a bit longer for the compatibility discussion [1] > >>>> and > >>>>>>>> decide the priority for this item afterwards. > >>>>>>>> > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> > >>>>>>>> Xintong > >>>>>>>> > >>>>>>>> > >>>>>>>> [1] https://lists.apache.org/list.html?dev@flink.apache.org > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, Jun 26, 2023 at 6:00 PM Chesnay Schepler < > >>>> ches...@apache.org > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> by-and-large I'm quite happy with the list of items. > >>>>>>>>> > >>>>>>>>> I'm curious as to why the "Disaggregated State Management" item > >>>> is > >>>>>>> marked > >>>>>>>>> as a must-have; will it require changes that break something? > >>>> What > >>>>>>>> prevents > >>>>>>>>> it from being added in 2.1? > >>>>>>>>> > >>>>>>>>> We may want to update the Java 17 item to "Make Java 17 the > >>>>> default, > >>>>>>> drop > >>>>>>>>> Java 8/11". Maybe even split it into a must-have "Drop Java 8" > >>>> and > >>>>> a > >>>>>>>>> nice-to-have "Drop Java 11"? > >>>>>>>>> > >>>>>>>>> "Move Calcite rules from Scala to Java": I would hope that this > >>>>> would > >>>>>>> be > >>>>>>>>> an entirely internal change, and could thus be an incremental > >>>>> process > >>>>>>>>> independent of major releases. > >>>>>>>>> What is the actual scale of this item; how much are we actually > >>>>>>>> re-writing? > >>>>>>>>> "Add MetricGroup#getLogicalScope": I'd raise this to a > >>>> must-have; i > >>>>>>> think > >>>>>>>>> I marked it down as nice-to-have only because it depends on > >>>> another > >>>>>>> item. > >>>>>>>>> The ProcessFunction API item is giving me the most headaches > >>>>> because > >>>>>>> it's > >>>>>>>>> very unclear what it actually entails; like is it an entirely > >>>>>> separate > >>>>>>>> API > >>>>>>>>> to DataStream (sounds like it is!) or an extension of DataStream. > >>>>> How > >>>>>>>> much > >>>>>>>>> will it share the internals with DataStream etc.; how does it > >>>>> relate > >>>>>> to > >>>>>>>> the > >>>>>>>>> Table API (w.r.t. switching APIs / what Table API uses > >>>> underneath). > >>>>>>>>> There are a few items I added as ideas which don't have a > >>>> priority > >>>>>> yet; > >>>>>>>>> would love to get some feedback on those. > >>>>>>>>> > >>>>>>>>> On 21/06/2023 08:41, Xintong Song wrote: > >>>>>>>>> > >>>>>>>>> Hi devs, > >>>>>>>>> > >>>>>>>>> As previously discussed in [1], we had been collecting work item > >>>>>>>> proposals > >>>>>>>>> for the 2.0 release until June 15th, on the wiki page [2]. > >>>>>>>>> > >>>>>>>>> - As we have passed the due date, I'd like to kindly remind > >>>>>> everyone > >>>>>>>> *not > >>>>>>>>> to add / remove items directly on the wiki page*. If needed, > >>>>>> please > >>>>>>>> post > >>>>>>>>> in this thread or reach out to the release managers instead. > >>>>>>>>> - I've reached out to some folks for clarifications about > >>>> their > >>>>>>>>> proposals. Some of them mentioned that they can not yet tell > >>>>>> whether > >>>>>>>> we > >>>>>>>>> should do an item or not, and would need more time / > >>>> discussions > >>>>>> to > >>>>>>>> make > >>>>>>>>> the decision. So I added a new symbol for items whose > >>>> priorities > >>>>>> are > >>>>>>>> `TBD`. > >>>>>>>>> Now it's time to collaboratively decide a minimum set of > >>>> must-have > >>>>>>> items. > >>>>>>>>> I've gone through the entire list of proposed items, and found > >>>> most > >>>>>> of > >>>>>>>> them > >>>>>>>>> make quite much sense. So I think an online sync might not be > >>>>>> necessary > >>>>>>>> for > >>>>>>>>> this. I'd like to go with this DISCUSS thread, where everyone can > >>>>>>> comment > >>>>>>>>> on how they think the list can be improved, followed by a VOTE to > >>>>>>>> formally > >>>>>>>>> make the decision. > >>>>>>>>> > >>>>>>>>> Any feedback and opinions, including but not limited to the > >>>>> following > >>>>>>>>> aspects, will be appreciated. > >>>>>>>>> > >>>>>>>>> - Important items that are missing from the list > >>>>>>>>> - Concerns regarding the listed items or their priorities > >>>>>>>>> > >>>>>>>>> Looking forward to your feedback. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> > >>>>>>>>> Xintong > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> [1] > >>>> > >> > https://lists.apache.org/list?dev@flink.apache.org:lte=1M:release%202.0%20status%20updates > >>>>>>>>> [2] > >>>> https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > >>>>>>>>> > >>>>>>>>> > >>>> > >>>> -- > >>>> Best regards, > >>>> Sergey > >>>> > >> > >> > >
