The FLIP LGTM. Thanks Xu, Jeyhun and Weijie for preparing this. +1 from my side
Best, Xintong On Wed, Dec 18, 2024 at 7:11 PM Yunfeng Zhou <flink.zhouyunf...@gmail.com> wrote: > Hi Xu Huang, > > I noticed your discussions mentioned that the identifier of watermarks > needs to be global across the whole job. Therefore, would it be better to > add to the FLIP a proposal to the naming convention of identifiers? For > example, the identifiers are encouraged to be named with its holding > module/connector as a prefix, like "runtime-backlog”. It is not a > compulsory requirement, and is more like a suggestion. > > Best, > Yufneng > > > 2024年12月17日 11:13，Xu Huang <huangxu.wal...@gmail.com> 写道： > > > > Thank you for participating in the discussion. > > > > @jrlee....@gmail.com <jrlee....@gmail.com> > >> If I have a large number of generalized watermarks that need to be > > created, where should they be declared? > > > > In the current design, the generalized watermark should be declared only > > once in the first user-defined function or source reader that needs to > > create and send it downstream. > > Furthermore, all watermark identifiers must remain unique within a single > > application. Once the upstream operator declares a generalized watermark, > > the downstream operator will be aware of it and will be able to process > it > > accordingly. > > > > Best, Xu Huang > > > > Junrui Lee <jrlee....@gmail.com> 于2024年12月16日周一 19:40写道： > > > >> Hi Xu Huang, > >> > >> Thanks for the proposal! > >> > >> I have a question: If I have a large number of generalized watermarks > that > >> need to be created, where should they be declared? Should they be > declared > >> only once in a single Source, or in all operators that need to send, > >> receive, and process them? > >> > >> Best regards, > >> > >> Junrui > >> > >> Xu Huang <huangxu.wal...@gmail.com> 于2024年12月10日周二 15:21写道： > >> > >>> Hi Devs, > >>> > >>> Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion > >> about > >>> FLIP-467: Introduce Generalized Watermarks [1]. > >>> > >>> Based on our findings, we recognize the need for specific events that > >>> require propagation and alignment across streams, functioning similarly > >> to > >>> watermarks. An example of this is the IsProcessingBacklog event > proposed > >> in > >>> FLIP-309 [2]. > >>> > >>> > >>> This has inspired us to create a more generalized watermark framework > >> that > >>> transcends traditional event time semantics. The generalized watermark > >>> framework allows users to define a variety of events that can be > emitted > >>> from the source or other operators, propagate through the streams, and > be > >>> received by downstream operators with aligned properties. With this > >>> abstraction, users and developers can design specialized events > according > >>> to their needs, such as EventTime watermark or idleness watermark > status. > >>> > >>> > >>> Note that this feature only worked for DataStream V2. > >>> > >>> For more details, please refer to FLIP-467 [1]. We look forward to your > >>> feedback. > >>> > >>> > >>> Best, > >>> > >>> Jeyhun Karimov, Weijie Guo and Xu Huang > >>> > >>> [1] > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks > >>> > >>> [2] > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog > >>> > >> > >
