Hi, Thanks for your reply! I have no other comments!
Best, Guowei On Mon, Feb 26, 2024 at 3:43 PM weijie guo <guoweijieres...@gmail.com> wrote: > Hi Guowei, > > thanks for your reply! > > > Do connectors and SQL currently have similar problems? > > - Connectors: > The APIs for FLIP-27 based source and Sink-V2 are currently in flink-core, > and we will gradually move them to flink-core-api. Anyway, connector should > be free of this problem. > > - SQL/Table: > > For pure SQL job, the user does not need to have any dependencies when > writing the SQL query string. > > But for the Table API, it relies on the original DataStream(i.e. DataStream > V1), so the implementation is not decoupled from the API. > > In the future, we will consider building the Table API on top of DataStream > API V2, which solves this problem also. > > > Best regards, > > Weijie > > > Guowei Ma <guowei....@gmail.com> 于2024年2月26日周一 14:37写道： > > > Hi,weijie > > > > Thank you very much to Weijie for proposing this series of improvements, > > especially the complete decoupling of user interface and implementation. > > This part is actually a very serious problem that disturbs downstream > users > > in the community. I hope this problem can be completely solved in the > > future. > > > > However, regarding the API decoupling part, I have a question: Do > > connectors and SQL currently have similar problems? If so, will similar > > methods be used to solve them? > > > > Best, > > Guowei > > > > > > On Tue, Feb 20, 2024 at 3:10 PM weijie guo <guoweijieres...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > Thanks for all the feedback. > > > > > > If there are no more comments, I would like to start the vote thread, > > > thanks again! > > > > > > Best regards, > > > > > > Weijie > > > > > > > > > Xintong Song <tonysong...@gmail.com> 于2024年1月30日周二 11:04写道： > > > > > > > Thanks for working on this, Weijie. > > > > > > > > The design flaws of the current DataStream API (i.e., V1) have been a > > > pain > > > > for a long time. It's great to see efforts going on trying to resolve > > > them. > > > > > > > > Significant changes to such an important and comprehensive set of > > public > > > > APIs deserves caution. From that perspective, the ideas of > introducing > > a > > > > new set of APIs that gradually replace the current one, splitting the > > > > introducing of the new APIs into many separate FLIPs, and making > > > > intermediate APIs @Experiemental until all of them are completed make > > > > great sense to me. > > > > > > > > Besides, the ideas of generalized watermark, execution hints sound > > quite > > > > interesting. Looking forward to more detailed discussions in the > > > > corresponding sub-FLIPs. > > > > > > > > +1 for the roadmap. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Tue, Jan 30, 2024 at 11:00 AM weijie guo < > guoweijieres...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi Wencong: > > > > > > > > > > > The Processing TimerService is currently > > > > > defined as one of the basic primitives, partly because it's > > understood > > > > that > > > > > you have to choose between processing time and event time. > > > > > The other part of the reason is that it needs to work based on the > > > task's > > > > > mailbox thread model to avoid concurrency issues. Could you clarify > > the > > > > > second > > > > > part of the reason? > > > > > > > > > > Since the processing logic of the operators takes place in the > > mailbox > > > > > thread, the processing timer's callback function must also be > > executed > > > in > > > > > the mailbox to ensure thread safety. > > > > > If we do not define the Processing TimerService as primitive, there > > is > > > no > > > > > way for the user to dispatch custom logic to the mailbox thread. > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Weijie > > > > > > > > > > > > > > > Xuannan Su <suxuanna...@gmail.com> 于2024年1月29日周一 17:12写道： > > > > > > > > > > > Hi Weijie, > > > > > > > > > > > > Thanks for driving the work! There are indeed many pain points in > > the > > > > > > current DataStream API, which are challenging to resolve with its > > > > > > existing design. It is a great opportunity to propose a new > > > DataStream > > > > > > API that tackles these issues. I like the way we've divided the > > FLIP > > > > > > into multiple sub-FLIPs; the roadmap is clear and comprehensible. > > +1 > > > > > > for the umbrella FLIP. I am eager to see the sub-FLIPs! > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 24, 2024 at 8:55 PM Wencong Liu < > liuwencle...@163.com> > > > > > wrote: > > > > > > > > > > > > > > Hi Weijie, > > > > > > > > > > > > > > > > > > > > > Thank you for the effort you've put into the DataStream API ! > By > > > > > > reorganizing and > > > > > > > redesigning the DataStream API, as well as addressing some of > the > > > > > > unreasonable > > > > > > > designs within it, we can enhance the efficiency of job > > development > > > > for > > > > > > developers. > > > > > > > It also allows developers to design more flexible Flink jobs to > > > meet > > > > > > business requirements. > > > > > > > > > > > > > > > > > > > > > I have conducted a comprehensive review of the DataStream API > > > design > > > > in > > > > > > versions > > > > > > > 1.18 and 1.19. I found quite a few functional defects in the > > > > DataStream > > > > > > API, such as the > > > > > > > lack of corresponding APIs in batch processing scenarios. In > the > > > > > > upcoming 1.20 version, > > > > > > > I will further improve the DataStream API in batch computing > > > > scenarios. > > > > > > > > > > > > > > > > > > > > > The issues existing in the old DataStream API (which can be > > > referred > > > > to > > > > > > as V1) can be > > > > > > > addressed from a design perspective in the initial version of > > V2. I > > > > > hope > > > > > > to also have the > > > > > > > opportunity to participate in the development of DataStream V2 > > and > > > > > make > > > > > > my contribution. > > > > > > > > > > > > > > > > > > > > > Regarding FLIP-408, I have a question: The Processing > > TimerService > > > is > > > > > > currently > > > > > > > defined as one of the basic primitives, partly because it's > > > > understood > > > > > > that > > > > > > > you have to choose between processing time and event time. > > > > > > > The other part of the reason is that it needs to work based on > > the > > > > > task's > > > > > > > mailbox thread model to avoid concurrency issues. Could you > > clarify > > > > the > > > > > > second > > > > > > > part of the reason? > > > > > > > > > > > > > > Best, > > > > > > > Wencong Liu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2023-12-26 14:42:20, "weijie guo" < > guoweijieres...@gmail.com> > > > > > wrote: > > > > > > > >Hi devs, > > > > > > > > > > > > > > > > > > > > > > > >I'd like to start a discussion about FLIP-408: [Umbrella] > > > Introduce > > > > > > > >DataStream API V2 [1]. > > > > > > > > > > > > > > > > > > > > > > > >The DataStream API is one of the two main APIs that Flink > > provides > > > > for > > > > > > > >writing data processing programs. As an API that was > introduced > > > > > > > >practically since day-1 of the project and has been evolved > for > > > > nearly > > > > > > > >a decade, we are observing more and more problems of it. > > > > Improvements > > > > > > > >on these problems require significant breaking changes, which > > > makes > > > > > > > >in-place refactor impractical. Therefore, we propose to > > introduce > > > a > > > > > > > >new set of APIs, the DataStream API V2, to gradually replace > the > > > > > > > >original DataStream API. > > > > > > > > > > > > > > > > > > > > > > > >The proposal to introduce a whole set new API is complex and > > > > includes > > > > > > > >massive changes. We are planning to break it down into > multiple > > > > > > > >sub-FLIPs for incremental discussion. This FLIP is only used > as > > an > > > > > > > >umbrella, mainly focusing on motivation, goals, and overall > > > > planning. > > > > > > > >That is to say, more design and implementation details will > be > > > > > > > >discussed in other FLIPs. > > > > > > > > > > > > > > > > > > > > > > > >Given that it's hard to imagine the detailed design of the new > > API > > > > if > > > > > > > >we're just talking about this umbrella FLIP, and we probably > > won't > > > > be > > > > > > > >able to give an opinion on it. Therefore, I have prepared two > > > > > > > >sub-FLIPs [2][3] at the same time, and the discussion of them > > will > > > > be > > > > > > > >posted later in separate threads. > > > > > > > > > > > > > > > > > > > > > > > >Looking forward to hearing from you, thanks! > > > > > > > > > > > > > > > > > > > > > > > >Best regards, > > > > > > > > > > > > > > > >Weijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >[1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2 > > > > > > > > > > > > > > > >[2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-409%3A+DataStream+V2+Building+Blocks%3A+DataStream%2C+Partitioning+and+ProcessFunction > > > > > > > > > > > > > > > > > > > > > > > >[3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-410%3A++Config%2C+Context+and+Processing+Timer+Service+of+DataStream+API+V2 > > > > > > > > > > > > > > > > > > > > >
