Hi Devs, Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion about FLIP-467: Introduce Generalized Watermarks [1].
Based on our findings, we recognize the need for specific events that require propagation and alignment across streams, functioning similarly to watermarks. An example of this is the IsProcessingBacklog event proposed in FLIP-309 [2]. This has inspired us to create a more generalized watermark framework that transcends traditional event time semantics. The generalized watermark framework allows users to define a variety of events that can be emitted from the source or other operators, propagate through the streams, and be received by downstream operators with aligned properties. With this abstraction, users and developers can design specialized events according to their needs, such as EventTime watermark or idleness watermark status. Note that this feature only worked for DataStream V2. For more details, please refer to FLIP-467 [1]. We look forward to your feedback. Best, Jeyhun Karimov, Weijie Guo and Xu Huang [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog
