A while back I wrote this slightly more elaborate extractor that will advance
the watermark independently after the stream is idle for a while:
https://github.com/aljoscha/flink/blob/6e4419e550caa0e5b162bc0d2ccc43f6b0b3860f/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functio
This depends on the requirements of your application.
Using the usual watermark generation strategies which are purely data
driven, a stream that does not produce data would not advance its
watermarks.
Not advancing the watermarks means that the program cannot make progress.
This might also be fin
Hi Fabian,
I want to extract timestamps from my event. However, the events stream can
be sparse at times (e.g. 2 days without any events).
What's the best strategy to create watermarks if I want real-time
processing of the events which enter the stream?
Jayant Ameta
On Thu, Jan 11, 2018 at 4:53 P
Another thing to point out is that watermarks are usually data-driven,
i.e., they depend on the timestamps of the events and not on the clock of
the machine.
Otherwise, you might observe a lot of late data, i.e., events with
timestamps smaller than the last watermark.
If you assign timestamps and
Thanks Gary,
I was only trying with a fixed set of events, so the Watermark was not
advancing, like you said.
Jayant Ameta
On Thu, Jan 11, 2018 at 3:36 PM, Gary Yao wrote:
> Hi Jayant,
>
> The difference is that the Watermarks from
> BoundedOutOfOrdernessTimestampExtractor are based on the gre
Hi Jayant,
The difference is that the Watermarks from
BoundedOutOfOrdernessTimestampExtractor are based on the greatest timestamp
of
all previous events. That is, if you do not receive new events, the
Watermark
will not advance. In contrast, your custom implementation of
AssignerWithPeriodicWaterm
Hi,
When using a BoundedOutOfOrdernessTimestampExtractor, the trigger is not
firing. However, the trigger fires when using custom timestamp extractor
with similar watermark.
Sample code below:
1.Assigner as anonymous class which works fine
AssignerWithPeriodicWatermarks> assigner = new
AssignerWi
