[
https://issues.apache.org/jira/browse/FLINK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40503:
-----------------------------------
Labels: pull-request-available (was: )
> WatermarksWithIdleness effective timeout is idleTimeout plus up to 3 probe
> intervals, contradicting the withIdleness documentation
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40503
> URL: https://issues.apache.org/jira/browse/FLINK-40503
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Reporter: Martijn Visser
> Priority: Major
> Labels: pull-request-available
>
> {{WatermarkStrategy#withIdleness}} documents: "if no records flow in a
> partition of a
> stream for that amount of time" the output is marked idle. The actual
> detection in
> {{WatermarksWithIdleness.IdlenessTimer}} anchors its countdown at the *first
> quiet
> periodic probe* (not at the last record) and uses a strict {{>}} comparison
> ({{WatermarksWithIdleness.java:140}}).
> With timeout T=10ms and probes every 5ms (event at t=4):
> probe t=5 observes activity and resets; t=10 is the first quiet probe and
> only *starts*
> the timer; t=15 elapsed 5ms; t=20 elapsed exactly 10ms fails the strict {{>}};
> *idle fires at t=25 — 21ms after the last event, 2.1× the configured timeout.*
> General form: idleness is declared between idleTimeout and idleTimeout + 3
> probe intervals after the last event - up to two intervals for the countdown
> to be anchored at the first quiet periodic probe, plus up to one more because
> the strict > comparison only fires at the next probe after the timeout is
> exceeded. The worst case is attained when the timeout is an exact multiple of
> the auto-watermark interval, as in the example (T=10ms, probes every 5ms →
> idle 21ms after the last event).
> This is distinct from FLINK-35886, which fixed *what time counts* toward the
> timeout
> (blocked/backpressured time); this issue is about *when the countdown starts
> and fires*.
> Proposed resolution: document the actual behavior (javadoc of
> {{withIdleness}} and the
> config documentation) rather than change the timing — silently changing
> detection timing
> has a regression track record (cf. the FLIP-471 fallout). Optionally,
> anchoring the
> timer at the last activity observation could be discussed separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
