Thanks for driving this, Jark. The current draft looks good to me. I think it is good to open a PR with it. And if there are other comments, we can discuss them during the PR review.
I also added a few minor comments in the draft regarding the feature radar. Those can also be discussed on the PR. Best, Xintong On Tue, Aug 15, 2023 at 11:15 AM Shammon FY <zjur...@gmail.com> wrote: > Hi Jark, > > Sounds good and I would love to, thanks! I will involve you and Xingtong > on the document after updating. > > Best, > Shammon FY > > > On Mon, Aug 14, 2023 at 10:39 PM Jark Wu <imj...@gmail.com> wrote: > >> Hi Shammon, >> >> Sure, could you help to draft a subsection about this in the google doc? >> >> Best, >> Jark >> >> 2023年8月14日 20:30，Shammon FY <zjur...@gmail.com> 写道： >> >> Thanks @Jark for driving the Flink Roadmap. >> >> As we discussed olap in the thread [1] and according to the suggestions >> from @Xingtong Song, could we add a subsection in `Towards Streaming >> Warehouses` or `Performance` that the short-lived query in Flink Session >> Cluster is one of the future directions for Flink? >> >> Best, >> Shammon FY >> >> On Mon, Aug 14, 2023 at 8:03 PM Jark Wu <imj...@gmail.com> wrote: >> >>> Thank you everyone for helping polish the roadmap [1]. >>> >>> I think I have addressed all the comments and we have included all >>> ongoing >>> parts of Flink. >>> Please feel free to take a last look. I'm going to prepare the pull >>> request >>> if there are no more concerns. >>> >>> Best, >>> Jark >>> >>> [1]: >>> >>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit >>> >>> On Sun, 13 Aug 2023 at 13:04, Yuan Mei <yuanmei.w...@gmail.com> wrote: >>> >>> > Sorry for taking so long >>> > >>> > I've added a section about Flink Disaggregated State Management >>> Evolution >>> > in the attached doc. >>> > >>> > I found some of the contents might be overlapped with the "large-scale >>> > streaming jobs". So that part might need some changes as well. >>> > >>> > Please let me know what you think. >>> > >>> > Best >>> > Yuan >>> > >>> > On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yuanmei.w...@gmail.com> >>> wrote: >>> > >>> > > Sorry have missed this email and respond a bit late. >>> > > >>> > > I will put a draft for the long-term vision for the state as well as >>> > > large-scale state support into the roadmap. >>> > > >>> > > Best >>> > > Yuan >>> > > >>> > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <imj...@gmail.com> wrote: >>> > > >>> > >> Hi Jiabao, >>> > >> >>> > >> Thank you for your suggestions. I have added them to the "Going >>> Beyond a >>> > >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" >>> > sections. >>> > >> >>> > >> Best, >>> > >> Jark >>> > >> >>> > >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao....@xtransfer.cn >>> > >> .invalid> >>> > >> wrote: >>> > >> >>> > >> > Thanks Jark and Martijn for driving this. >>> > >> > >>> > >> > There are two suggestions about the Table API: >>> > >> > >>> > >> > - Add the JSON type to adapt to the no sql database type. >>> > >> > - Remove changelog normalize operator for upsert stream. >>> > >> > >>> > >> > >>> > >> > Best, >>> > >> > Jiabao >>> > >> > >>> > >> > >>> > >> > > 2023年7月13日 下午3:49，Jark Wu <imj...@gmail.com> 写道： >>> > >> > > >>> > >> > > Hi all, >>> > >> > > >>> > >> > > Sorry for taking so long back here. >>> > >> > > >>> > >> > > Martijn and I have drafted the first version of the updated >>> roadmap, >>> > >> > > including the updated feature radar reflecting the current >>> state of >>> > >> > > different components. >>> > >> > > >>> > >> > >>> > >> >>> > >>> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit >>> > >> > > >>> > >> > > Feel free to leave comments in the thread or the document. >>> > >> > > We may miss mentioning something important, so your help in >>> > enriching >>> > >> > > the content is greatly appreciated. >>> > >> > > >>> > >> > > Best, >>> > >> > > Jark & Martijn >>> > >> > > >>> > >> > > >>> > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <j...@ververica.com.invalid >>> > >>> > >> wrote: >>> > >> > > >>> > >> > >> Hi Jark, >>> > >> > >> >>> > >> > >> Fair enough. Let's do it like you suggested. Thanks! >>> > >> > >> >>> > >> > >> Best regards, >>> > >> > >> Jing >>> > >> > >> >>> > >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <imj...@gmail.com> >>> wrote: >>> > >> > >> >>> > >> > >>> Hi Jing, >>> > >> > >>> >>> > >> > >>> This thread is for discussing the roadmap for versions 1.18, >>> 2.0, >>> > >> and >>> > >> > >> even >>> > >> > >>> more. >>> > >> > >>> One of the outcomes of this discussion will be an updated >>> version >>> > of >>> > >> > the >>> > >> > >>> current roadmap. >>> > >> > >>> Let's work together on refining the roadmap in this thread. >>> > >> > >>> >>> > >> > >>> Best, >>> > >> > >>> Jark >>> > >> > >>> >>> > >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge >>> <j...@ververica.com.invalid> >>> > >> > wrote: >>> > >> > >>> >>> > >> > >>>> Hi Jark, >>> > >> > >>>> >>> > >> > >>>> Thanks for driving it! For point 2, since we are developing >>> 1.18 >>> > >> now, >>> > >> > >>>> does it make sense to update the roadmap this time while we >>> are >>> > >> > >> releasing >>> > >> > >>>> 1.18? This discussion thread will be focusing on the Flink >>> 2.0 >>> > >> > roadmap, >>> > >> > >>> as >>> > >> > >>>> you mentioned previously. WDYT? >>> > >> > >>>> >>> > >> > >>>> Best regards, >>> > >> > >>>> Jing >>> > >> > >>>> >>> > >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <imj...@gmail.com> >>> wrote: >>> > >> > >>>> >>> > >> > >>>>> Hi all, >>> > >> > >>>>> >>> > >> > >>>>> Martijn and I would like to initiate a discussion on the >>> Flink >>> > >> > >> roadmap, >>> > >> > >>>>> which should cover the project's long-term roadmap and the >>> > regular >>> > >> > >>> update >>> > >> > >>>>> mechanism. >>> > >> > >>>>> >>> > >> > >>>>> Xintong has already started a discussion about Flink 2.0 >>> > planning. >>> > >> > >> One >>> > >> > >>> of >>> > >> > >>>>> the points raised in that discussion is that we should have >>> a >>> > >> > >>> high-level >>> > >> > >>>>> discussion of the roadmap to present where the project is >>> > heading >>> > >> > >>> (which >>> > >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning). >>> > >> Moreover, >>> > >> > >>> the >>> > >> > >>>>> roadmap on the Flink website [1] hasn't been updated for >>> half a >>> > >> year, >>> > >> > >>> and >>> > >> > >>>>> the last update was for the feature radar for the 1.15 >>> release. >>> > It >>> > >> > >> has >>> > >> > >>>> been >>> > >> > >>>>> 2 years since the community discussed Flink's overall >>> roadmap. >>> > >> > >>>>> >>> > >> > >>>>> I would like to raise two topics for discussion: >>> > >> > >>>>> >>> > >> > >>>>> 1. The new roadmap. This should be an updated version of the >>> > >> current >>> > >> > >>>>> roadmap[1]. >>> > >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap. >>> > >> > >>>>> >>> > >> > >>>>> To make the first topic discussion more efficient, Martijn >>> and I >>> > >> > >>>> volunteer >>> > >> > >>>>> to summarize the ongoing big things of different components >>> and >>> > >> > >>> present a >>> > >> > >>>>> roadmap draft to the community in the next few weeks. This >>> > should >>> > >> be >>> > >> > >> a >>> > >> > >>>> good >>> > >> > >>>>> starting point for a more detailed discussion. >>> > >> > >>>>> >>> > >> > >>>>> Regarding the regular update mechanism, there was a >>> proposal in >>> > a >>> > >> > >>> thread >>> > >> > >>>>> [2] three years ago to make the release manager responsible >>> for >>> > >> > >>> updating >>> > >> > >>>>> the roadmap. However, it appears that this was not >>> documented >>> > as a >>> > >> > >>>> release >>> > >> > >>>>> management task [3], and the roadmap update wasn't >>> performed for >>> > >> > >>> releases >>> > >> > >>>>> 1.16 and 1.17. >>> > >> > >>>>> >>> > >> > >>>>> In my opinion, making release managers responsible for >>> keeping >>> > the >>> > >> > >>>> roadmap >>> > >> > >>>>> up to date is a good idea. Specifically, release managers of >>> > >> release >>> > >> > >> X >>> > >> > >>>> can >>> > >> > >>>>> kick off the roadmap update at the beginning of release X, >>> which >>> > >> can >>> > >> > >>> be a >>> > >> > >>>>> joint task with collecting a feature list [4]. Additionally, >>> > >> release >>> > >> > >>>>> managers of release X-1 can help verify and remove the >>> > >> accomplished >>> > >> > >>> items >>> > >> > >>>>> from the roadmap and update the feature radar. >>> > >> > >>>>> >>> > >> > >>>>> What do you think? Do you have other ideas? >>> > >> > >>>>> >>> > >> > >>>>> Best, >>> > >> > >>>>> Jark & Martijn >>> > >> > >>>>> >>> > >> > >>>>> [1]: https://flink.apache.org/roadmap.html >>> > >> > >>>>> [2]: >>> > >> > >> >>> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv >>> > >> > >>>>> [3]: >>> > >> > >>>>> >>> > >> > >>>> >>> > >> > >>> >>> > >> > >> >>> > >> > >>> > >> >>> > >>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management >>> > >> > >>>>> [4]: >>> > >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release >>> > >> > >>>>> >>> > >> > >>>> >>> > >> > >>> >>> > >> > >> >>> > >> > >>> > >> > >>> > >> >>> > > >>> > >>> >> >>
