Tianji Li created FLINK-40495:
---------------------------------
Summary: Limit concurrent shared-state disposal to prevent
JobManager I/O starvation
Key: FLINK-40495
URL: https://issues.apache.org/jira/browse/FLINK-40495
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 2.2.0
Reporter: Tianji Li
Attachments: flink-jobmanager-pools-after.png,
flink-jobmanager-pools-before.png
When there are lots of old checkpoint files on S3 to be deleted by the
JobManager, it may wait for a long time before it could have change to send
checkpoint barriers. This can cause long checkpoint durations unnecessarily and
inconsistently (it only happens when deleting many old files).
In our recently, the JM sometimes waited around 330 seconds before it could
send checkpoint barriers, even if the configured checkpoint duration is 180
seconds. Long checkpoint durations lead to very high processing delays when TMs
are recycled and hence TMs recreation and reprocessing have to be done.
The root cause is the shared thread-pool (jobmanager.io-pool) used by both of
these operations. In the attached flink-jobmanager-pools-before.png file,
please find what is the current way that the pool is shared. We have an
internal fix (see in the flink-jobmanager-pools-before.png) that limits the S3
cleanup through a configuration.
!flink-jobmanager-pools-after.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
