Hi, Sorry for the late response. +1 for deprecating it.
It would be even better to just remove it from the code base, but would require a little bit of investigation in the kinesis connector [1], if this feature can be safely removed from the kinesis connector in favour of the generic watermark alignment. Best, Piotrek [1] https://github.com/apache/flink-connector-aws/blob/38aafb44d3a8200e4ff41d87e0780338f40da258/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/JobManagerWatermarkTracker.java#L40 pon., 4 gru 2023 o 11:54 Zhanghao Chen <zhanghao.c...@outlook.com> napisał(a): > Hi Benchao， > > I think part of the reason is that a general global coordination mechanism > is complex and hence subject to some internals changes in the future. > Instead of directly exposing the full mechanism to users, it might be > better to expose some well-defined subset of the feature set to users. > > I'm also ccing the email to Piotr and David for their suggestions on this. > > Best, > Zhanghao Chen > ________________________________ > From: Benchao Li <libenc...@apache.org> > Sent: Monday, November 27, 2023 13:03 > To: dev@flink.apache.org <dev@flink.apache.org> > Subject: Re: [DISCUSS] FLIP-395: Deprecate Global Aggregator Manager > > +1 for the idea. > > Currently OperatorCoordinator is still marked as @Internal, shouldn't > it be a public API already? > > Besides, GlobalAggregatorManager supports coordination between > different operators, but OperatorCoordinator only supports > coordination within one operator. And CoordinatorStore introduced in > FLINK-24439 opens the door for multi operators. Again, should it also > be a public API too? > > Weihua Hu <huweihua....@gmail.com> 于2023年11月27日周一 11:05写道： > > > > Thanks Zhanghao for driving this FLIP. > > > > +1 for this. > > > > Best, > > Weihua > > > > > > On Mon, Nov 20, 2023 at 5:49 PM Zhanghao Chen <zhanghao.c...@outlook.com > > > > wrote: > > > > > Hi all, > > > > > > I'd like to start a discussion of FLIP-395: Deprecate Global Aggregator > > > Manager [1]. > > > > > > Global Aggregate Manager was introduced in [2] to support event time > > > synchronization across sources and more generally, coordination of > parallel > > > tasks. AFAIK, this was only used in the Kinesis source for an early > version > > > of watermark alignment. Operator Coordinator, introduced in FLIP-27, > > > provides a more powerful and elegant solution for that need and is > part of > > > the new source API standard. FLIP-217 further provides a complete > solution > > > for watermark alignment of source splits on top of the Operator > Coordinator > > > mechanism. Furthermore, Global Aggregate Manager manages state in > JobMaster > > > object, causing problems for adaptive parallelism changes [3]. > > > > > > Therefore, I propose to deprecate the use of Global Aggregate Manager, > > > which can improve the maintainability of the Flink codebase without > > > compromising its functionality. > > > > > > Looking forward to your feedbacks, thanks. > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-395%3A+Deprecate+Global+Aggregator+Manager > > > [2] https://issues.apache.org/jira/browse/FLINK-10886 > > > [3] https://issues.apache.org/jira/browse/FLINK-31245 > > > > > > Best, > > > Zhanghao Chen > > > > > > > -- > > Best, > Benchao Li >
