Martijn Visser created FLINK-40502:
--------------------------------------
Summary: scan.watermark.idle-timeout is silently ignored for
connectors without watermark pushdown; SOURCE_WATERMARK() ignores idle-timeout
entirely
Key: FLINK-40502
URL: https://issues.apache.org/jira/browse/FLINK-40502
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Martijn Visser
{{table.exec.source.idle-timeout}} / {{scan.watermark.idle-timeout}} resolve
differently
depending on plan shape:
# *Watermark pushdown plans* ({{SupportsWatermarkPushDown}}): global option
honored,
per-table option / OPTIONS hint honored, hint wins
({{WatermarkPushDownSpec#calculateIdleTimeoutMillis}}; pinned by the expected
plans of
{{PushWatermarkIntoTableSourceScanRuleTest#testIdleSourceWithOptions/Hint}} —
{{idletimeout=[60000]}} beats a global 1000ms).
# *Standalone WatermarkAssigner plans* (connector without pushdown support —
e.g.
datagen, filesystem, JDBC-style scans): {{scan.watermark.idle-timeout}} is
*silently
ignored*. {{StreamPhysicalWatermarkAssigner.translateToExecNode}} drops hints
and table
options; {{StreamExecWatermarkAssigner}} reads only the global
{{TABLE_EXEC_SOURCE_IDLE_TIMEOUT}} and its exec node carries no idle-timeout
field at
all. No data path from table options to {{WatermarkAssignerOperatorFactory}}
exists.
The DDL still validates cleanly because {{FactoryUtil}} unconditionally marks
all
{{scan.watermark.*}} keys as consumed for every connector factory
({{FactoryUtil.java:1069-1071}}); nothing warns at validation or planning
time.
# *SOURCE_WATERMARK() plans*: both options are ignored entirely — the pushdown
rule
returns before the idle-timeout is read, and {{SourceWatermarkSpec}} has no
idle-timeout member.
# Additionally, compiled JSON plans persist no idle-timeout (the exec node
serializes
only {{watermarkExpr}} and {{rowtimeFieldIndex}}), so a restored compiled plan
re-resolves idleness from the *restore-time* global configuration.
FLIP-296 (FLINK-31540) deliberately scoped the option to pushdown-capable
sources; the
only notice is a hint box in the time-attributes documentation. FLINK-20947 was
the
mirror bug (global option dropped in the pushdown path).
Proposed fix: route {{WatermarkParams}} into {{StreamExecWatermarkAssigner}} so
the
per-table option works uniformly (and persists in compiled plans); at minimum,
log a
warning when {{scan.watermark.*}} options are set on a non-pushdown source and
document
the scoping in the option description.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
