Hi David, If watermarks are ignored how do Consecutive windowed operations [1] work? I’m just trying to understand in which scenarios I need to assign timestamps and watermarks, or if it’s enough if I do it once near the beginning of the DAG (assuming the source doesn’t do it).
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#consecutive-windowed-operations Regards, Alexis. From: David Morávek <d...@apache.org> Sent: Donnerstag, 2. Dezember 2021 17:26 To: Alexis Sarda-Espinosa <alexis.sarda-espin...@microfocus.com> Cc: user@flink.apache.org Subject: Re: Watermark behavior when connecting streams Hi Alexis, please take a look at AbstractStreamOperator [1] for details how the watermark is calculate for TwoInputOperator. It uses pretty much the same approach as for with the single input one (it simply takes a minimum). For watermark re-assignment, we ignore input watermark unless it's Long.MAX_LONG (this happens on shutdown, eg. savepoint + drain). You can see more details by looking at the TimestampsAndWatermarksOperator [2]. [1] https://github.com/apache/flink/blob/release-1.14.0/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L608 [2] https://github.com/apache/flink/blob/release-1.14.0/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/TimestampsAndWatermarksOperator.java#L124 Best, D. On Wed, Dec 1, 2021 at 9:49 AM Alexis Sarda-Espinosa <alexis.sarda-espin...@microfocus.com<mailto:alexis.sarda-espin...@microfocus.com>> wrote: Hi everyone, Based on what I know, a single operator with parallelism > 1 checks the watermarks from all its streams and uses the smallest one out of the non-idle streams. My first question is whether watermarks are forwarded as long as a different watermark strategy is not applied downstream? For example, will my stream keep its watermarks even after windowing + processing? The second question is what happens with watermarks after connecting 2 streams, specifically these 2 possibilities: - One stream was watermarks but the other one doesn’t. - Both streams have watermarks. Regards, Alexis.
