Hi Benchao， I think part of the reason is that a general global coordination mechanism is complex and hence subject to some internals changes in the future. Instead of directly exposing the full mechanism to users, it might be better to expose some well-defined subset of the feature set to users.
I'm also ccing the email to Piotr and David for their suggestions on this. Best, Zhanghao Chen ________________________________ From: Benchao Li <libenc...@apache.org> Sent: Monday, November 27, 2023 13:03 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS] FLIP-395: Deprecate Global Aggregator Manager +1 for the idea. Currently OperatorCoordinator is still marked as @Internal, shouldn't it be a public API already? Besides, GlobalAggregatorManager supports coordination between different operators, but OperatorCoordinator only supports coordination within one operator. And CoordinatorStore introduced in FLINK-24439 opens the door for multi operators. Again, should it also be a public API too? Weihua Hu <huweihua....@gmail.com> 于2023年11月27日周一 11:05写道： > > Thanks Zhanghao for driving this FLIP. > > +1 for this. > > Best, > Weihua > > > On Mon, Nov 20, 2023 at 5:49 PM Zhanghao Chen <zhanghao.c...@outlook.com> > wrote: > > > Hi all, > > > > I'd like to start a discussion of FLIP-395: Deprecate Global Aggregator > > Manager [1]. > > > > Global Aggregate Manager was introduced in [2] to support event time > > synchronization across sources and more generally, coordination of parallel > > tasks. AFAIK, this was only used in the Kinesis source for an early version > > of watermark alignment. Operator Coordinator, introduced in FLIP-27, > > provides a more powerful and elegant solution for that need and is part of > > the new source API standard. FLIP-217 further provides a complete solution > > for watermark alignment of source splits on top of the Operator Coordinator > > mechanism. Furthermore, Global Aggregate Manager manages state in JobMaster > > object, causing problems for adaptive parallelism changes [3]. > > > > Therefore, I propose to deprecate the use of Global Aggregate Manager, > > which can improve the maintainability of the Flink codebase without > > compromising its functionality. > > > > Looking forward to your feedbacks, thanks. > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-395%3A+Deprecate+Global+Aggregator+Manager > > [2] https://issues.apache.org/jira/browse/FLINK-10886 > > [3] https://issues.apache.org/jira/browse/FLINK-31245 > > > > Best, > > Zhanghao Chen > > -- Best, Benchao Li
