Hi, As far as I am concerned, it would be great to build two top level categories for Flink 2.0 release.
1. Future - mainly new big features or architecture improvements to achieve visions like streamhouse. This makes the 2.0 release be the 2.0 instead of 1.x release, i.e. the real intention of 2.0 release with a significant upgrade. 2. History - clean up tech debts, take care of histories, or whatever you want to name it. The main goal of this category is to take the 2.0 release opportunity (since we have strong intention to do it mentioned above) to perform breaking changes, i.e. remove deprecated APIs and even modules, upgrade APIs without thinking more about backwards compatibilities, etc. This is kind of a buy-one-get-one benefit. In order to "get-one"(History), we should, first of all, "buy-one"(Future). Best regards, Jing On Fri, Apr 28, 2023 at 9:57 AM Xintong Song <tonysong...@gmail.com> wrote: > Thanks all for the positive feedback. > > So far, it seems to me that the differences of opinions are mainly focused > on whether we should include non-breaking features in the 2.0 release. > > There seems to be no objections to: > 1. Starting to plan for the 2.0 release, with a rough timeline towards mid > next year > 2. Becket, Jark, Martijn and Xintong as the release managers > 3. Initiating a project roadmap discussion > > I'll leave this discussion open for a bit longer. Also, next week is public > holidays in China (I don't know if it's also in other countries). After the > holidays and if there's no objections, we'll assemble the release > management team as discussed, and try to figure out a proper way for the > roadmap discussion next. > > Best, > > Xintong > > > > On Fri, Apr 28, 2023 at 3:43 PM Xintong Song <tonysong...@gmail.com> > wrote: > > > @Weike, > > > > Thanks for the suggestion. I think it makes sense to provide a longer > > supporting period for the last 1.x release. > > > > @David, > > > > I can see the benefit of focusing on breaking changes and API clean-ups > > only, so the community can be more focused and possibly deliver the 2.0 > > release earlier. However, I can also understand that it might be > > disappointing for some people (admittedly, a little bit for me as well) > if > > the 2.0 release contains only clean-ups but no other significant > user-aware > > improvements. My personal opinion would be not to prevent people from > > trying to get new features into this release, but would not block the > > release of such features unless breaking changes are involved. > > > > @Martijn, > > > > Thanks for sharing your ideas. Glad to see that things you've listed have > > a lot in common with what we put in our list. I believe that's a good > > signal that we share similar opinions on what is good and important for > the > > project and the release. > > > > @Sai, > > > > Welcome to the community. And thanks for offering helps. > > > > At the moment, this discussion is only happening in this mailing list. We > > may consider setting up online meetings or dedicated slack channels in > > future. And if so, the information will also be posted in the mailing > list. > > > > Best, > > > > Xintong > > > > > > > > On Fri, Apr 28, 2023 at 2:19 PM Saichandrasekar TM < > > saichandrase...@gmail.com> wrote: > > > >> Hi All, > >> > >> Awesome...I see this as a great opportunity for newcomers like me to > >> contribute. > >> > >> Is this discussion happening in a slack or discord forum too? If so, pls > >> include me. > >> > >> Thanks, > >> Sai > >> > >> On Fri, Apr 28, 2023 at 2:55 AM Martijn Visser < > martijnvis...@apache.org> > >> wrote: > >> > >> > Hi all, > >> > > >> > I think the proposal is a good starting point. We should aim to make > >> Flink > >> > a unified data processing, cloud friendly / cloud native technology, > >> with > >> > proper low-level and high-level interfaces (DataStream API, Table API, > >> > SQL). I think it would make a lot of sense that we write down a vision > >> for > >> > Flink for the long term. That would also mean sharing and discussing > >> more > >> > insights and having conversations around some of the long-term > direction > >> > from the proposal. > >> > > >> > In order to achieve that vision, I believe that we need a Flink 2.0 > >> which I > >> > consider a long overdue clean-up. That version should be the > foundation > >> for > >> > Flink that allows the above mentioned vision to become actual > proposals > >> and > >> > implementations. > >> > > >> > As a foundation in Flink 2.0, I would be inclined to say it should be: > >> > > >> > - Remove all deprecated APIs, including the DataSet API, Scala API, > >> > Queryable State, legacy Source and Sink implementations, legacy SQL > >> > functions etc. > >> > - Add support for Java 17 and 21, make 17 the default (given that the > >> next > >> > Java LTS, 21, is released in September this year and the timeline is > >> set of > >> > 2024) > >> > - Drop support for Java 8 and 11 > >> > - Refactor the configuration layer > >> > - Refactor the DataStream API, such as: > >> > ** Having a coherent and well designed API > >> > ** Decouple the API into API-only modules, so no more cyclic > >> dependencies > >> > and leaking of non-APIs, including Kryo > >> > ** Reorganize APIs and modules > >> > > >> > I think these are some of the must-haves. Curious about the thoughts > of > >> the > >> > community. > >> > > >> > Thanks, Martijn > >> > > >> > Op do 27 apr. 2023 om 10:16 schreef David Morávek <d...@apache.org> > >> > > >> > > Hi, > >> > > > >> > > Great to see this topic moving forward; I agree it's long overdue. > >> > > > >> > > I keep thinking about 2.0 as a chance to eliminate things that > didn't > >> > work, > >> > > make the feature set denser, and fix rough edges and APIs that hold > us > >> > > back. > >> > > > >> > > Some items in the doc (Key Features section) don't tick these boxes > >> for > >> > me, > >> > > as they could also be implemented in the 1x branch. We should > consider > >> > > whether we need a backward incompatible release to introduce each > >> > feature. > >> > > This should help us to keep the discussion more focused. > >> > > > >> > > Best, > >> > > D. > >> > > > >> > > > >> > > On Wed, Apr 26, 2023 at 2:33 PM DONG Weike <kyled...@connect.hku.hk > > > >> > > wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > It is thrilling to see the foreseeable upcoming rollouts of Flink > >> 2.x > >> > > > releases, and I believe that this roadmap can take Flink to the > next > >> > > stage > >> > > > of a top-of-notch unified streaming & batch computing engine. > >> > > > > >> > > > Given that all of the existing user programs are written and run > in > >> > Flink > >> > > > 1.x versions as for now, and some of them are very complex and > rely > >> on > >> > > > various third-party connectors written with legacy APIs, one thing > >> > that I > >> > > > have concerns about is if, one day in the future, the community > >> decides > >> > > > that new features are only given to 2.x releases, could the last > >> > release > >> > > of > >> > > > Flink 1.x be converted as an LTS version (backporting severe bug > >> fixes > >> > > and > >> > > > critical security patches), so that existing users could have > enough > >> > time > >> > > > to wait for third-party connectors to upgrade, test their programs > >> on > >> > the > >> > > > Flink APIs, and avoid sudden loss of community support. > >> > > > > >> > > > Just my two cents : ) > >> > > > > >> > > > Best, > >> > > > Weike > >> > > > > >> > > > ________________________________ > >> > > > 发件人: Xintong Song <tonysong...@gmail.com> > >> > > > 发送时间: 2023年4月26日 20:01 > >> > > > 收件人: dev <dev@flink.apache.org> > >> > > > 主题: Re: [DISCUSS] Planning Flink 2.0 > >> > > > > >> > > > @Chesnay > >> > > > > >> > > > > >> > > > > Technically this implies that every minor release may contain > >> > breaking > >> > > > > changes, which is exactly what users don't want. > >> > > > > >> > > > > >> > > > It's not necessary to introduce the breaking chagnes immediately > >> upon > >> > > > reaching the minimum guaranteed stable time. If there are multiple > >> > > changes > >> > > > waiting for the stable time, we can still gather them in 1 minor > >> > release. > >> > > > But I see your point, from the user's perspective, the mechanism > >> does > >> > not > >> > > > provide any guarantees for the compatibility of minor releases. > >> > > > > >> > > > What problems to do you see in creating major releases every N > >> years? > >> > > > > > >> > > > > >> > > > It might not be concrete problem, but I'm a bit concerned by the > >> > > > uncertainty. I assume N should not be too small, e.g., at least 3. > >> I'd > >> > > > expect the decision to ship a major release would be made based on > >> > > > comprehensive considerations over the situations at that time. > >> Making a > >> > > > decision now that we would ship a major release 3 years later > seems > >> a > >> > bit > >> > > > agressive to me. > >> > > > > >> > > > We need to figure out what this release means for connectors > >> > > > > compatibility-wise. > >> > > > > > >> > > > > >> > > > +1 > >> > > > > >> > > > > >> > > > > What process are you thinking of for deciding what breaking > >> changes > >> > to > >> > > > > make? The obvious choice would be FLIPs, but I'm worried that > this > >> > will > >> > > > > overload the mailing list / wiki for lots of tiny changes. > >> > > > > > >> > > > > >> > > > This should be a community decision. What I have in mind would be: > >> (1) > >> > > > collect a wish list on wiki, (2) schedule a series of online > >> meetings > >> > > (like > >> > > > the release syncs) to get an agreed set of must-have items, (3) > >> develop > >> > > and > >> > > > polish the detailed plans of items via FLIPs, and (4) if the plan > >> for a > >> > > > must-have item does not work out then go back to (2) for an > update. > >> I'm > >> > > > also open to other opinions. > >> > > > > >> > > > Would we wait a few months for people to prepare/agree on changes > >> so we > >> > > > > reduce the time we need to merge things into 2 branches? > >> > > > > > >> > > > > >> > > > That's what I had in mind. Hopefully after 1.18. > >> > > > > >> > > > @Max > >> > > > > >> > > > When I look at > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit > >> > > > > , I'm a bit skeptical we will even be able to reach all these > >> goals. > >> > I > >> > > > > think we have to prioritize and try to establish a deadline. > >> > Otherwise > >> > > we > >> > > > > will end up never releasing 2.0. > >> > > > > >> > > > > >> > > > Sorry for the confusion. I should have explain this more clearly. > We > >> > are > >> > > > not planning to finish all the items in the list. It's more like a > >> > > > brainstorm, a list of candidates. We are also expecting to collect > >> more > >> > > > ideas from the community. And after collecting the ideas, we > should > >> > > > prioritize them and decide on a subset of must-have items, > following > >> > the > >> > > > consensus decision making. > >> > > > > >> > > > +1 on Flink 2.0 by May 2024 (not a hard deadline but I think > having > >> a > >> > > > > deadline helps). > >> > > > > > >> > > > > >> > > > I agree that having a deadline helps. I proposed mid 2024, which > is > >> > > similar > >> > > > to but not as explicit as what you proposed. We may start with > >> having a > >> > > > deadline for deciding the must-have items (e.g., by the end of > >> June). > >> > > That > >> > > > should make it easier for estimating the overall time needed for > >> > > preparing > >> > > > the release. > >> > > > > >> > > > Best, > >> > > > > >> > > > Xintong > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Apr 26, 2023 at 6:57 PM Gyula Fóra <gyf...@apache.org> > >> wrote: > >> > > > > >> > > > > +1 to everything Max said. > >> > > > > > >> > > > > Gyula > >> > > > > > >> > > > > On Wed, 26 Apr 2023 at 11:42, Maximilian Michels < > m...@apache.org> > >> > > wrote: > >> > > > > > >> > > > > > Thanks for starting the discussion, Jark and Xingtong! > >> > > > > > > >> > > > > > Flink 2.0 is long overdue. In the past, the expectations for > >> such a > >> > > > > > release were unreasonably high. I think everybody had a > >> different > >> > > > > > understanding of what exactly the criteria were. This led to > >> > > releasing > >> > > > > > 18 minor releases for the current major version. > >> > > > > > > >> > > > > > What I'm most excited about for Flink 2.0 is removal of > baggage > >> > that > >> > > > > > Flink has accumulated over the years: > >> > > > > > > >> > > > > > - Removal of Scala, deprecated interfaces, unmaintained > >> libraries > >> > and > >> > > > > > APIs (DataSet) > >> > > > > > - Consolidation of configuration > >> > > > > > - Merging of multiple scheduler implementations > >> > > > > > - Ability to freely combine batch / streaming tasks in the > >> runtime > >> > > > > > > >> > > > > > When I look at > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit > >> > > > > > , I'm a bit skeptical we will even be able to reach all these > >> > goals. > >> > > I > >> > > > > > think we have to prioritize and try to establish a deadline. > >> > > Otherwise > >> > > > > > we will end up never releasing 2.0. > >> > > > > > > >> > > > > > +1 on Flink 2.0 by May 2024 (not a hard deadline but I think > >> > having a > >> > > > > > deadline helps). > >> > > > > > > >> > > > > > -Max > >> > > > > > > >> > > > > > > >> > > > > > On Wed, Apr 26, 2023 at 10:08 AM Chesnay Schepler < > >> > > ches...@apache.org> > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > > /Instead of defining compatibility guarantees as "this > API > >> > won't > >> > > > > > > change in all 1.x/2.x series", what if we define it as "this > >> API > >> > > > won't > >> > > > > > > change in the next 2/3 years"./ > >> > > > > > > > >> > > > > > > I can see some benefits to this approach (all APIs having a > >> fixed > >> > > > > > > minimum lifetime) but it's just gonna be difficult to > >> > communicate. > >> > > > > > > Technically this implies that every minor release may > contain > >> > > > breaking > >> > > > > > > changes, which is exactly what users don't want. > >> > > > > > > > >> > > > > > > What problems to do you see in creating major releases > every N > >> > > years? > >> > > > > > > > >> > > > > > > > /IIUC, the milestone releases are a breakdown of the 2.0 > >> > > release, > >> > > > > > > while we are free to introduce breaking changes between > them. > >> And > >> > > you > >> > > > > > > suggest using longer-living feature branches to keep the > >> master > >> > > > branch > >> > > > > > > in a releasable state (in terms of milestone releases). Am I > >> > > > > > > understanding it correctly?/ > >> > > > > > > > >> > > > > > > I think you got the general idea. There are a lot of details > >> to > >> > be > >> > > > > > > ironed out though (e.g., do we release connectors for each > >> > > > > > milestone?...). > >> > > > > > > > >> > > > > > > Conflicts in the long-lived branches are certainly a > concern, > >> > but I > >> > > > > > > think those will be inevitable. Right now I'm not _too_ > >> worried > >> > > about > >> > > > > > > them, at least based on my personal wish-list. > >> > > > > > > Maybe the milestones could even help with that, as we could > >> > > > > preemptively > >> > > > > > > decide on an order for certain changes that have a high > >> chance of > >> > > > > > > conflicting with each other? > >> > > > > > > I guess we could do that anyway. > >> > > > > > > Maybe we should explicitly evaluate how invasive a change is > >> (in > >> > > > > > > relation to other breaking changes!) and manage things > >> > accordingly > >> > > > > > > > >> > > > > > > > >> > > > > > > Other thoughts: > >> > > > > > > > >> > > > > > > We need to figure out what this release means for connectors > >> > > > > > > compatibility-wise. The current rules for which versions a > >> > > connector > >> > > > > > > must support don't cover major releases at all. > >> > > > > > > (This depends a bit on the scope of 2.0; if we add binary > >> > > > compatibility > >> > > > > > > to Public APIs and promote a few Evolving ones then > >> compatibility > >> > > > > across > >> > > > > > > minor releases becomes trivial) > >> > > > > > > > >> > > > > > > What process are you thinking of for deciding what breaking > >> > changes > >> > > > to > >> > > > > > > make? The obvious choice would be FLIPs, but I'm worried > that > >> > this > >> > > > will > >> > > > > > > overload the mailing list / wiki for lots of tiny changes. > >> > > > > > > > >> > > > > > > Provided that we agree on doing 2.0, when would we cut the > 2.0 > >> > > > branch? > >> > > > > > > Would we wait a few months for people to prepare/agree on > >> changes > >> > > so > >> > > > we > >> > > > > > > reduce the time we need to merge things into 2 branches? > >> > > > > > > > >> > > > > > > On 26/04/2023 05:51, Xintong Song wrote: > >> > > > > > > > Thanks all for the positive feedback. > >> > > > > > > > > >> > > > > > > > @Martijn > >> > > > > > > > > >> > > > > > > > If we want to have that roadmap, should we consolidate > this > >> > into > >> > > a > >> > > > > > > >> dedicated Confluence page over storing it in a Google > doc? > >> > > > > > > >> > >> > > > > > > > Having a dedicated wiki page is definitely a good way for > >> the > >> > > > roadmap > >> > > > > > > > discussion. I haven't created one yet because it's still a > >> > > proposal > >> > > > > to > >> > > > > > have > >> > > > > > > > such roadmap discussion. If the community agrees with our > >> > > proposal, > >> > > > > the > >> > > > > > > > release manager team can decide how they want to drive and > >> > track > >> > > > the > >> > > > > > > > roadmap discussion. > >> > > > > > > > > >> > > > > > > > @Chesnay > >> > > > > > > > > >> > > > > > > > We should discuss how regularly we will ship major > releases > >> > from > >> > > > now > >> > > > > > on. > >> > > > > > > >> Let's avoid again making breaking changes because we > >> "gotta do > >> > > it > >> > > > > now > >> > > > > > > >> because 3.0 isn't happening anytime soon". (e.g., every 2 > >> > years > >> > > or > >> > > > > > > >> something) > >> > > > > > > > > >> > > > > > > > I'm not entirely sure about shipping major releases > >> regularly. > >> > > But > >> > > > I > >> > > > > do > >> > > > > > > > agree that we may want to avoid the situation that > "breaking > >> > > > changes > >> > > > > > can > >> > > > > > > > only happen now, or no idea when". Instead of defining > >> > > > compatibility > >> > > > > > > > guarantees as "this API won't change in all 1.x/2.x > series", > >> > what > >> > > > if > >> > > > > we > >> > > > > > > > define it as "this API won't change in the next 2/3 > years". > >> > That > >> > > > > should > >> > > > > > > > allow us to incrementally iterate the APIs. > >> > > > > > > > > >> > > > > > > > E.g., in 2.a, all APIs annotated as `@Stable` will be > >> > guaranteed > >> > > > > > compatible > >> > > > > > > > until 2 years after 2.a is shipped, and in 2.b if the API > is > >> > > still > >> > > > > > > > annotated `@Stable` it extends the compatibility guarantee > >> to 2 > >> > > > years > >> > > > > > after > >> > > > > > > > 2.b is shipped. To remove an API, we would need to mark it > >> as > >> > > > > > `@Deprecated` > >> > > > > > > > and wait for 2 years after the last release in which it > was > >> > > marked > >> > > > > > > > `@Stable`. > >> > > > > > > > > >> > > > > > > > My thinking goes rather in the area of defining Milestone > >> > > releases, > >> > > > > > each > >> > > > > > > >> Milestone targeting specific changes. > >> > > > > > > >> > >> > > > > > > > I'm trying to understand what you are suggesting here. > IIUC, > >> > the > >> > > > > > milestone > >> > > > > > > > releases are a breakdown of the 2.0 release, while we are > >> free > >> > to > >> > > > > > introduce > >> > > > > > > > breaking changes between them. And you suggest using > >> > > longer-living > >> > > > > > feature > >> > > > > > > > branches to keep the master branch in a releasable state > (in > >> > > terms > >> > > > of > >> > > > > > > > milestone releases). Am I understanding it correctly? > >> > > > > > > > > >> > > > > > > > I haven't thought this through. My gut feeling is this > might > >> > be a > >> > > > > good > >> > > > > > > > direction to go, in terms of keeping things organized. The > >> risk > >> > > is > >> > > > > the > >> > > > > > cost > >> > > > > > > > of merging feature branches and rebasing feature branches > >> after > >> > > > other > >> > > > > > > > features are merged. That depends on how close the > features > >> are > >> > > > > > related to > >> > > > > > > > each other. E.g., reorganization of the project modules > and > >> > > > > > dependencies > >> > > > > > > > may change the project structure a lot, which may > >> significantly > >> > > > > affect > >> > > > > > most > >> > > > > > > > of the feature branches. Maybe we can identify such > >> > > > widely-affecting > >> > > > > > > > changes and perform them at the beginning or end of the > >> release > >> > > > > cycle. > >> > > > > > > > > >> > > > > > > > Best, > >> > > > > > > > > >> > > > > > > > Xintong > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Wed, Apr 26, 2023 at 8:23 AM ConradJam< > >> jam.gz...@gmail.com> > >> > > > > wrote: > >> > > > > > > > > >> > > > > > > >> Thanks Xintong and Jark for kicking off the great > >> discussion! > >> > > > > > > >> > >> > > > > > > >> I checked the list carefully. The plans are detailed and > >> most > >> > of > >> > > > the > >> > > > > > > >> problems are covered > >> > > > > > > >> Some of the ideas Chesnay mentioned, I think we should > >> iterate > >> > > in > >> > > > > > > >> small steps and collect feedback in time > >> > > > > > > >> Looking forward to the start of the work of Flink2.0, I > am > >> > > willing > >> > > > > to > >> > > > > > > >> provide assistance ~ > >> > > > > > > >> > >> > > > > > > >> Xintong Song<tonysong...@gmail.com> 于2023年4月25日周二 > >> 19:10写道： > >> > > > > > > >>> Hi everyone, > >> > > > > > > >>> > >> > > > > > > >>> I'd like to start a discussion on planning for a Flink > 2.0 > >> > > > release. > >> > > > > > > >>> > >> > > > > > > >>> AFAIK, in the past years this topic has been mentioned > >> from > >> > > time > >> > > > to > >> > > > > > time, > >> > > > > > > >>> in mailing lists, jira tickets and offline discussions. > >> > > However, > >> > > > > few > >> > > > > > > >>> concrete steps have been taken, due to the significant > >> > > > > determination > >> > > > > > and > >> > > > > > > >>> efforts it requires and distractions from other > >> prioritized > >> > > > > focuses. > >> > > > > > > >> After > >> > > > > > > >>> a series of offline discussions in the recent weeks, > with > >> > folks > >> > > > > > mostly > >> > > > > > > >> from > >> > > > > > > >>> our team internally as well as a few from outside > Alibaba > >> / > >> > > > > Ververica > >> > > > > > > >>> (thanks for insights from Becket and Robert), we believe > >> it's > >> > > > time > >> > > > > to > >> > > > > > > >> kick > >> > > > > > > >>> this off in the community. > >> > > > > > > >>> > >> > > > > > > >>> Below are some of our thoughts about the 2.0 release. > >> Looking > >> > > > > > forward to > >> > > > > > > >>> your opinions and feedback. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> ## Why plan for release 2.0? > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Flink 1.0.0 was released in March 2016. In the past 7 > >> years, > >> > > many > >> > > > > new > >> > > > > > > >>> features have been added and the project has become > >> different > >> > > > from > >> > > > > > what > >> > > > > > > >> it > >> > > > > > > >>> used to be. So what is Flink now? What will it become in > >> the > >> > > next > >> > > > > 3-5 > >> > > > > > > >>> years? What do we think of Flink's position in the > >> industry? > >> > We > >> > > > > > believe > >> > > > > > > >>> it's time to rethink these questions, and draw a roadmap > >> > > towards > >> > > > > > another > >> > > > > > > >>> milestone, a milestone that worths a new major release. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> In addition, we are still providing backwards > >> compatibility > >> > > > (maybe > >> > > > > > not > >> > > > > > > >>> perfectly but largely) with APIs that we designed and > >> claimed > >> > > > > stable > >> > > > > > 7 > >> > > > > > > >>> years ago. While such backwards compatibility helps > users > >> to > >> > > > stick > >> > > > > > with > >> > > > > > > >> the > >> > > > > > > >>> latest Flink releases more easily, it sometimes, and > more > >> and > >> > > > more > >> > > > > > over > >> > > > > > > >>> time, also becomes a burden for maintenance and a > >> limitation > >> > > for > >> > > > > new > >> > > > > > > >>> features and improvements. It's probably time to have a > >> > > > > comprehensive > >> > > > > > > >>> review and clean-up over all the public APIs. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Furthermore, next year is the 10th year for Flink as an > >> > Apache > >> > > > > > project. > >> > > > > > > >>> Flink joined the Apache incubator in April 2014, and > >> became a > >> > > > > > top-level > >> > > > > > > >>> project in December 2014. That makes 2024 a perfect time > >> for > >> > > > > > bringing out > >> > > > > > > >>> the release 2.0 milestone. And for such a major release, > >> we'd > >> > > > > expect > >> > > > > > it > >> > > > > > > >>> takes one year or even longer to prepare for, which > means > >> we > >> > > > > probably > >> > > > > > > >>> should start now. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> ## What should we focus on in release 2.0? > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> - Roadmap discussion - How do we define and position > >> > Flink > >> > > > for > >> > > > > > now and > >> > > > > > > >>> in future? This is probably something we lacked. I > >> > believe > >> > > > some > >> > > > > > > >> people have > >> > > > > > > >>> thought about it, but at least it's not explicitly > >> > > discussed > >> > > > > and > >> > > > > > > >> aligned in > >> > > > > > > >>> the community. Ideally, the 2.0 release should be a > >> > result > >> > > of > >> > > > > the > >> > > > > > > >> roadmap. > >> > > > > > > >>> - Breaking changes - Important improvements, > bugfixes, > >> > > > > technical > >> > > > > > debts > >> > > > > > > >>> that involve breaking of API backwards > compatibility, > >> > which > >> > > > can > >> > > > > > only > >> > > > > > > >> be > >> > > > > > > >>> carried out in major releases. > >> > > > > > > >>> - With breaking API changes, we may need multiple > >> > > > > > 2.0-alpha/beta > >> > > > > > > >>> versions to collect feedback. > >> > > > > > > >>> - Key features - Significant features and > improvements > >> > > (e.g., > >> > > > > > new user > >> > > > > > > >>> stories, architectural upgrades) that may change how > >> > users > >> > > > use > >> > > > > > Flink > >> > > > > > > >> and > >> > > > > > > >>> its position in the industry. Some items from this > >> > category > >> > > > may > >> > > > > > also > >> > > > > > > >>> involve API breaking changes or significant behavior > >> > > changes. > >> > > > > > > >>> - There are also opinions that we should stay > >> focused > >> > as > >> > > > > much > >> > > > > > as > >> > > > > > > >>> possible on the breaking changes only. > Incremental > >> / > >> > > > > > non-breaking > >> > > > > > > >>> improvements and features, or anything that can > be > >> > added > >> > > > in > >> > > > > > 2.x > >> > > > > > > >> minor > >> > > > > > > >>> releases, should not block the 2.0 release. > >> > > > > > > >>> > >> > > > > > > >>> It might be better to discuss the detailed technical > items > >> > > later > >> > > > in > >> > > > > > > >> another > >> > > > > > > >>> thread, to keep the current discussion focused on the > >> overall > >> > > > > > proposal, > >> > > > > > > >> and > >> > > > > > > >>> to leave time for all parties to think about their > >> technical > >> > > > plans. > >> > > > > > For > >> > > > > > > >>> your reference, I've attached a preliminary list of work > >> > items > >> > > > > > proposed > >> > > > > > > >> by > >> > > > > > > >>> Alibaba / Ververica [1]. Note that the listed items are > >> still > >> > > > being > >> > > > > > > >>> carefully evaluated and prioritized, and may change in > >> > future. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> ## How do we manage the release? > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> #### Release Process > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> We'd expect the release process for Flink 2.0 to be > >> different > >> > > > from > >> > > > > > the > >> > > > > > > >> 1.x > >> > > > > > > >>> releases. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> A major difference is that, we think the timeline-based > >> > release > >> > > > > > > >> management > >> > > > > > > >>> may not be suitable. The idea behind the timeline-based > >> > > approach > >> > > > is > >> > > > > > that > >> > > > > > > >> we > >> > > > > > > >>> can have more frequent releases and deliver completed > >> > features > >> > > to > >> > > > > > users > >> > > > > > > >>> earlier, while incompleted features can be postponed to > >> the > >> > > next > >> > > > > > release > >> > > > > > > >>> which won't be too late with the short release cycle. > >> > However, > >> > > > for > >> > > > > > > >> breaking > >> > > > > > > >>> changes that can only take place in major releases, the > >> price > >> > > for > >> > > > > > > >> missing a > >> > > > > > > >>> release is too high. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Alternatively, we probably should discuss and agree on a > >> list > >> > > of > >> > > > > > > >> must-have > >> > > > > > > >>> work items. That doesn't mean keep postponing the > release > >> > upon > >> > > a > >> > > > > few > >> > > > > > > >>> delayed features. In fact, we would need to closely > >> monitor > >> > the > >> > > > > > progress > >> > > > > > > >> of > >> > > > > > > >>> the must-have items during the entire release cycle, > >> making > >> > > sure > >> > > > > > they are > >> > > > > > > >>> taken care of by contributors with enough expertise and > >> > > > capacities. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> #### Timeline > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> The release cycle should be decided according to the > >> feature > >> > > > list, > >> > > > > > > >>> especially the must-have items that we plan to do in the > >> > > release. > >> > > > > > > >> However, > >> > > > > > > >>> a target feature freeze date would still be helpful when > >> > making > >> > > > the > >> > > > > > plan, > >> > > > > > > >>> so that we don't pack too many things into the release. > We > >> > > > propose > >> > > > > > to aim > >> > > > > > > >>> for a feature freeze around mid 2024, so that in case > >> > must-have > >> > > > > > items are > >> > > > > > > >>> delayed, we still have a good chance to make the release > >> > happen > >> > > > by > >> > > > > > the > >> > > > > > > >> end > >> > > > > > > >>> of 2024. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> #### Branch > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> A longer release cycle also means we probably should > keep > >> > > shiping > >> > > > > > the 1.x > >> > > > > > > >>> releases while preparing for the 2.0 release. We may > cut a > >> > > > > release-1 > >> > > > > > from > >> > > > > > > >>> master, on which we can keep developing and release 1.x > >> > > releases. > >> > > > > The > >> > > > > > > >>> version on the master branch will then become > >> '2.0-SNAPSHOT'. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> #### Release Manager > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Given the new and to-be-explored release process, longer > >> > cycle > >> > > > and > >> > > > > > higher > >> > > > > > > >>> synchronization requirements, we'd expect the 2.0 > release > >> to > >> > be > >> > > > > more > >> > > > > > > >>> challenging than previous 1.x releases. Therefore, we'd > >> like > >> > to > >> > > > > > propose > >> > > > > > > >> to > >> > > > > > > >>> assemble a release management team with 4-5 experienced > >> PMC > >> > > > > members. > >> > > > > > Jark > >> > > > > > > >>> and I would like to volunteer as 2 of the release > >> managers. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Looking forward to your thoughts. > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> Best, > >> > > > > > > >>> > >> > > > > > > >>> Jark & Xintong > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> [1] > >> > > > > > > >>> > >> > > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > >> > > > > > > >> > >> > > > > > > >> -- > >> > > > > > > >> Best > >> > > > > > > >> > >> > > > > > > >> ConradJam > >> > > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> Thanks, > >> Sai > >> *+91 917 623 3379* > >> > > >
