[
https://issues.apache.org/jira/browse/FLINK-40093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-40093:
--------------------------------------
Assignee: Efrat Levitan
> Race condition between watermark alignment and idleness detection locks split
> in idle
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40093
> URL: https://issues.apache.org/jira/browse/FLINK-40093
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Reporter: Efrat Levitan
> Assignee: Efrat Levitan
> Priority: Major
>
> Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't
> marked idle while paused. However with low idleness timeout (observed with
> 1s) + low allowed WM drift, a race condition could cause paused splits to
> never resume though they have records:
> # A split becomes paused due to too advanced records.
> # pauseOrResumeSplits pauses the split.
> # pauseOrResumeSplits reaches to pause the split idleness clock but is
> \{idlenessTimeout} too late, and the split becomes idle.
> # The watermark advances but the split is excluded from the watermark
> alignment check due to its idleness.
> # More records arrive but the split is paused at the connector level so they
> aren't processed, nor seen by watermarkGenerator so it still considers the
> split idle
> A possible fix could be preserving the part where idle splits are excluded
> from alignment pause (to not override their idle status) while allowing
> alignment check to resume splits even if they are currently idle. They are
> considered idle until they emit the next record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
