Efrat19 opened a new pull request, #28689:
URL: https://github.com/apache/flink/pull/28689
## What is the purpose of the change
Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't
marked idle while paused. However with low idleness timeout (observed with 1s)
+ low allowed WM drift, a race condition could cause paused splits to never
resume though they have records:
1. A split becomes paused due to too advanced records.
2. pauseOrResumeSplits pauses the split.
3. pauseOrResumeSplits reaches to pause the split idleness clock but is
{idlenessTimeout} too late, and the split becomes idle.
4. The watermark advances but the split is excluded from the watermark
alignment check due to its idleness.
5. More records arrive but the split is paused at the connector level so
they aren't processed, nor seen by watermarkGenerator so it still considers the
split idle
The PR aims to fix it by preserving the part where idle splits are excluded
from alignment pause (to not override their idle status) while allowing
alignment check to resume splits even if they are currently idle. They are
considered idle until they emit the next record.
## Brief change log
- Allow idle splits to be resumed by alignment check
## Verifying this change
This change added tests and can be verified as follows:
-
`SourceOperatorSplitWatermarkAlignmentTest#testPausedIdleSplitsCanBeResumedByAlignmentCheck`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
I used `claude-sonnet-5` for the unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
