Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi Xu,Thank you. Regards. AnilOn Sunday, January 5, 2025 at 06:31:15 PM PST, Xu Huang wrote: Hi, Anil Maybe the watermark alignment mechanism on DataStream V1 can solve your problem, it can align the watermark of SourceSplit. please refer to the documentation [1][2]. And we don't provi

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi, Junrui Thanks for your question, Source is able to send event-time watermark. It can be sent via SourceReaderContext as mentioned in FLIP-467. Example is as follows: > sourceReaderContext.emitWatermark(EventTimeExtension.EVENT_TIME_WATERMARK_DECLARATION.newWatermark(eventTime)) We will add it

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi, Xu Thanks for your work, I noticed that in this FLIP, event-time watermark is created and sent through a separate WatermarkGenerator. I would like to know if there is support for Source to send event-time watermark? Best, Junrui Xu Huang 于2025年1月6日周一 10:31写道： > Hi, Anil > > Maybe the water

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi, Anil Maybe the watermark alignment mechanism on DataStream V1 can solve your problem, it can align the watermark of SourceSplit. please refer to the documentation [1][2]. And we don't provide this feature on this FLIP, this feature will be in our future planning. [1] https://cwiki.apache.org/

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi Xu,Thanks for the response.I am currently using Spark Streaming to process data from Kafka in microbatches, writing each microbatch's data to a dedicated prefix in S3. Since Spark Streaming is lazy, it processes data only when a microbatch is created or triggered, leaving resources idle unt

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi, Anil I don't understand what you mean by Global Watermark, are you trying to have all Sources emit a special event with the same epoch at the same time? Is there a specific user case for this question? Happy new year! Best, Xu Huang Anil Dasari 于2025年1月3日周五 14:02写道： > Hello XU,Happy new

Re: [DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hello XU,Happy new year. Thank you for FLIP-499 and FLIP-467. I tried to split/chunk streams based by fixed timestamp intervals and route them to the appropriate destination. A few months ago, I evaluated the following options and found that Flink currently lacks direct support for a global wat

[DISCUSS] FLIP-499: Support Event Time by Generalized Watermark in DataStream V2

Hi Devs, Weijie Guo and I would like to initiate a discussion about FLIP-499: Support Event Time by Generalized Watermark in DataStream V2 [1]. Event time is a fundamental