Hi Ray,

in principle, watermarks can be injected anywhere in a stream by calling
DataStream.assignTimestampsAndWatermarks().
However, timestamps are usually injected as soon as possible after a stream
in ingested (before the first shuffle). The reason is that watermarks
depend on the order of events (and their timestamps) in the stream. While
Flink guarantees the order of events within a partition, a shuffle
interleaves events of different partitions in an unpredictable way such
that it is not possible to reason about the order of timestamps afterwards.

The most common way to inject watermarks is directly inside of a
SourceFunction or with a TimestampAssigner before the first shuffle.

Best, Fabian

2017-06-09 0:46 GMT+02:00 Ray Ruvinskiy <[email protected]>:

> I’m trying to build a mental model of how watermarks get injected into the
> stream. Suppose I have a stream with a parallel source, and I’m running a
> cluster with multiple task managers. Does each parallel source reader
> inject watermarks, which are then forwarded to downstream consumers and
> shuffled between task managers? Or are watermarks created after the
> shuffle, when the stream records reach their destined task manager and
> right before they’re processed by the operator?
>
>
>
> Thanks,
>
>
>
> Ray
>

Reply via email to