Hi Xu Huang, “INTERNAL_RUNTIME_BACKLOG” sounds good to me. But I’m not sure whether it is proper to treat connector watermarks as internal, or which connectors should be regarded as internal. I’m okay with it to add the naming strategy to the FLIP now before we reached an agreement on the details here, to hopefully involve more discussions from other people.
Besides, your suggestion also reminds me that the FLIP might need to clarify whether the identifiers are case-sensitive. Best, Yunfeng > 2024年12月19日 11:54，Xu Huang <huangxu.wal...@gmail.com> 写道： > > Hi，@Yunfeng >> add to the FLIP a proposal to the naming convention of identifiers > > Thank you for your suggestion! It's a great solution to remind users that > the identifier should be unique. > In my opinion, the Flink internal identifiers could be named as > "INTERNAL_MODULE_XXX," such as "INTERNAL_RUNTIME_BACKLOG" or > "INTERNAL_CONNECTOR_KAFKA_IDLE." > > What do you think? I will incorporate this naming strategy into the FLIP if > you're okay with it. > > Thanks for your suggestion again! > > Best, > Xu Huang > > Yunfeng Zhou <flink.zhouyunf...@gmail.com> 于2024年12月18日周三 19:10写道： > >> Hi Xu Huang, >> >> I noticed your discussions mentioned that the identifier of watermarks >> needs to be global across the whole job. Therefore, would it be better to >> add to the FLIP a proposal to the naming convention of identifiers? For >> example, the identifiers are encouraged to be named with its holding >> module/connector as a prefix, like "runtime-backlog”. It is not a >> compulsory requirement, and is more like a suggestion. >> >> Best, >> Yufneng >> >>> 2024年12月17日 11:13，Xu Huang <huangxu.wal...@gmail.com> 写道： >>> >>> Thank you for participating in the discussion. >>> >>> @jrlee....@gmail.com <jrlee....@gmail.com> >>>> If I have a large number of generalized watermarks that need to be >>> created, where should they be declared? >>> >>> In the current design, the generalized watermark should be declared only >>> once in the first user-defined function or source reader that needs to >>> create and send it downstream. >>> Furthermore, all watermark identifiers must remain unique within a single >>> application. Once the upstream operator declares a generalized watermark, >>> the downstream operator will be aware of it and will be able to process >> it >>> accordingly. >>> >>> Best, Xu Huang >>> >>> Junrui Lee <jrlee....@gmail.com> 于2024年12月16日周一 19:40写道： >>> >>>> Hi Xu Huang, >>>> >>>> Thanks for the proposal! >>>> >>>> I have a question: If I have a large number of generalized watermarks >> that >>>> need to be created, where should they be declared? Should they be >> declared >>>> only once in a single Source, or in all operators that need to send, >>>> receive, and process them? >>>> >>>> Best regards, >>>> >>>> Junrui >>>> >>>> Xu Huang <huangxu.wal...@gmail.com> 于2024年12月10日周二 15:21写道： >>>> >>>>> Hi Devs, >>>>> >>>>> Jeyhun Karimov, Weijie Guo and I would like to initiate a discussion >>>> about >>>>> FLIP-467: Introduce Generalized Watermarks [1]. >>>>> >>>>> Based on our findings, we recognize the need for specific events that >>>>> require propagation and alignment across streams, functioning similarly >>>> to >>>>> watermarks. An example of this is the IsProcessingBacklog event >> proposed >>>> in >>>>> FLIP-309 [2]. >>>>> >>>>> >>>>> This has inspired us to create a more generalized watermark framework >>>> that >>>>> transcends traditional event time semantics. The generalized watermark >>>>> framework allows users to define a variety of events that can be >> emitted >>>>> from the source or other operators, propagate through the streams, and >> be >>>>> received by downstream operators with aligned properties. With this >>>>> abstraction, users and developers can design specialized events >> according >>>>> to their needs, such as EventTime watermark or idleness watermark >> status. >>>>> >>>>> >>>>> Note that this feature only worked for DataStream V2. >>>>> >>>>> For more details, please refer to FLIP-467 [1]. We look forward to your >>>>> feedback. >>>>> >>>>> >>>>> Best, >>>>> >>>>> Jeyhun Karimov, Weijie Guo and Xu Huang >>>>> >>>>> [1] >>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks >>>>> >>>>> [2] >>>>> >>>>> >>>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog >>>>> >>>> >> >>
