[
https://issues.apache.org/jira/browse/FLINK-40235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099481#comment-18099481
]
Jing Hung commented on FLINK-40235:
-----------------------------------
Hi [~WeiqingYang],
Thanks for the pointer. I did look at FLINK-38477 while researching this, and
my read is that they're related but slightly different bugs:
- FLINK-38477: a subtask that never receives a split finishes immediately and
emits `Long.MAX_VALUE`, which over-advances the watermark.
- This one: a subtask that had splits, then all finish mid-life (e.g. after a
Kinesis reshard) while the job keeps running, but the subtask itself isn't
finished and might get a new split later, so it shouldn't just be excluded
like a finished input.
If 38477 is heading toward a FLIP, might be worth covering this case there
too. Do you have a link to that FLIP discussion? Happy to fold this scenario in
as an additional case to cover if that's a better fit.
Thanks,
Jing
> FLIP-27 source subtask stalls the watermark after all its splits finish
> ------------------------------------------------------------------------
>
> Key: FLINK-40235
> URL: https://issues.apache.org/jira/browse/FLINK-40235
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common
> Affects Versions: 1.19.3, 2.1.0, 1.20.3
> Reporter: Jing Hung
> Priority: Major
>
> *Problem Summary*
> In an unbounded FLIP-27 source, if a subtask finishes all of its assigned
> splits while the job keeps running, it's left with zero active splits but is
> not finished. Its watermark freezes at the last emitted value and holds back
> the global (operator-min) watermark indefinitely.
> *Reproduction*
> - Unbounded Kinesis (or Kafka w/ rebalancing) source, `withIdleness(...)` set.
> - Parallelism high enough that a subtask can reach zero active splits after a
> reshard.
> - Observe: operator `currentOutputWatermark` stalls; the affected subtask
> reports a frozen watermark while others advance.
> *Note*
> - withIdleness() does not help: idleness is tracked per-split (each split has
> its own timer, advanced only while the split is live), so with zero splits
> there is no generator left to ever mark the subtask idle.
> - FLIP-27 has no reader/enumerator-level equivalent of the legacy
> SourceContext.markAsTemporarilyIdle(), so there is currently no way to signal
> that such a subtask should stop holding back the watermark. Reintroducing
> that capability may be one direction for a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
