Martijn Visser created FLINK-40503:
--------------------------------------
Summary: WatermarksWithIdleness effective timeout is idleTimeout
plus up to 2× the periodic probe interval, contradicting the withIdleness
documentation
Key: FLINK-40503
URL: https://issues.apache.org/jira/browse/FLINK-40503
Project: Flink
Issue Type: Bug
Components: API / Core
Reporter: Martijn Visser
{{WatermarkStrategy#withIdleness}} documents: "if no records flow in a
partition of a
stream for that amount of time" the output is marked idle. The actual detection
in
{{WatermarksWithIdleness.IdlenessTimer}} anchors its countdown at the *first
quiet
periodic probe* (not at the last record) and uses a strict {{>}} comparison
({{WatermarksWithIdleness.java:140}}).
With timeout T=10ms and probes every 5ms (event at t=4):
probe t=5 observes activity and resets; t=10 is the first quiet probe and only
*starts*
the timer; t=15 elapsed 5ms; t=20 elapsed exactly 10ms fails the strict {{>}};
*idle fires at t=25 — 21ms after the last event, 2.1× the configured timeout.*
General form: last event + T + up to 2 probe intervals — one to anchor the
timer, one
extra whenever the timeout is an exact multiple of the auto-watermark interval,
which is
the common production configuration. The same overshoot exists in every
consumer of
{{IdlenessTimer}} (DataStream V1 sources, {{assignTimestampsAndWatermarks}},
DataStream V2).
This is distinct from FLINK-35886, which fixed *what time counts* toward the
timeout
(blocked/backpressured time); this issue is about *when the countdown starts
and fires*.
Proposed resolution: document the actual behavior (javadoc of {{withIdleness}}
and the
config documentation) rather than change the timing — silently changing
detection timing
has a regression track record (cf. the FLIP-471 fallout). Optionally, anchoring
the
timer at the last activity observation could be discussed separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
