Thank you for participating in the discussion. @jrlee....@gmail.com <jrlee....@gmail.com> > If I have a large number of generalized watermarks that need to be created, where should they be declared?
In the current design, the generalized watermark should be declared only once in the first user-defined function or source reader that needs to create and send it downstream. Furthermore, all watermark identifiers must remain unique within a single application. Once the upstream operator declares a generalized watermark, the downstream operator will be aware of it and will be able to process it accordingly. Best, Xu Huang Junrui Lee <jrlee....@gmail.com> 于2024年12月16日周一 19:40写道： > Hi Xu Huang, > > Thanks for the proposal! > > I have a question: If I have a large number of generalized watermarks that > need to be created, where should they be declared? Should they be declared > only once in a single Source, or in all operators that need to send, > receive, and process them? > > Best regards, > > Junrui > > Xu Huang <huangxu.wal...@gmail.com> 于2024年12月10日周二 15:21写道： > > > Hi Devs, > > > > Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion > about > > FLIP-467: Introduce Generalized Watermarks [1]. > > > > Based on our findings, we recognize the need for specific events that > > require propagation and alignment across streams, functioning similarly > to > > watermarks. An example of this is the IsProcessingBacklog event proposed > in > > FLIP-309 [2]. > > > > > > This has inspired us to create a more generalized watermark framework > that > > transcends traditional event time semantics. The generalized watermark > > framework allows users to define a variety of events that can be emitted > > from the source or other operators, propagate through the streams, and be > > received by downstream operators with aligned properties. With this > > abstraction, users and developers can design specialized events according > > to their needs, such as EventTime watermark or idleness watermark status. > > > > > > Note that this feature only worked for DataStream V2. > > > > For more details, please refer to FLIP-467 [1]. We look forward to your > > feedback. > > > > > > Best, > > > > Jeyhun Karimov, Weijie Guo and Xu Huang > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks > > > > [2] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog > > >
