Thanks for the proposal Aljoscha. This is a very useful unification. We
have considered this FLIP already in the interfaces for FLIP-95 [1] and
look forward to update to the new unified watermark generators once
FLIP-126 has been accepted.
Regards,
Timo
[1] https://github.com/apache/flink/pull/11692
On 20.04.20 18:10, Aljoscha Krettek wrote:
Hi Everyone!
We would like to start a discussion on "FLIP-126: Unify (and separate)
Watermark Assigners" [1]. This work was started by Stephan in an
experimental branch. I expanded on that work to provide a PoC for the
changes proposed in this FLIP: [2].
Currently, we have two different flavours of Watermark
Assigners: AssignerWithPunctuatedWatermarks
and AssignerWithPeriodicWatermarks. Both of them extend
from TimestampAssigner. This means that sources that want to support
watermark assignment/extraction in the source need to support two
separate interfaces, we have two operator implementations for the
different flavours. Also, this makes features such as generic support
for idleness detection more complicated to implemented because we again
have to support two types of watermark assigners.
In this FLIP we propose two things:
Unify the Watermark Assigners into one Interface WatermarkGenerator
Separate this new interface from the TimestampAssigner
The motivation for the first is to simplify future implementations and
code duplication. The motivation for the second point is again code
deduplication, most assigners currently have to extend from some base
timestamp extractor or duplicate the extraction logic, or users have to
override an abstract method of the watermark assigner to provide the
timestamp extraction logic.
Additionally, we propose to add a generic wrapping WatermarkGenerator
that provides idleness detection, i.e. it can mark a stream/partition as
idle if no data arrives after a configured timeout.
The "unify and separate" part refers to the fact that we want to unify
punctuated and periodic assigners but at the same time split the
timestamp assigner from the watermark generator.
Please find more details in the FLIP [1]. Looking forward to
your feedback.
Best,
Aljoscha
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners
[2] https://github.com/aljoscha/flink/tree/stephan-event-time
