[
https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-33090.
----------------------------------
Fix Version/s: 1.19.0
Release Note: Now when disposing of no longer needed checkpoints, every
state handle/state file will be disposed in parallel by the ioExecutor, vastly
improving disposing speed of a single checkpoint (for large checkpoints the
disposal time can be improved from 10 minutes to < 1 minute) . The old
behaviour can be restored by setting `state.checkpoint.cleaner.parallel-mode`
to `false`.
Resolution: Fixed
merged commit 4de0f76 into apache:master
> CheckpointsCleaner clean individual checkpoint states in parallel
> -----------------------------------------------------------------
>
> Key: FLINK-33090
> URL: https://issues.apache.org/jira/browse/FLINK-33090
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.17.1
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Currently CheckpointsCleaner clean multiple checkpoints in parallel with
> JobManager's ioExecutor, however each checkpoint states is cleaned
> sequentially. With thousands of StateObjects to clean this can take long time
> on some checkpoint storage, if longer than the checkpoint interval this
> prevents new checkpointing.
> The proposal is to use the same ioExecutor to clean up each checkpoints
> states in parallel as well. From my local testing, with default settings for
> ioExecutor thread pool for xK state files this can reduce clean up time from
> 10 minutes to <1 minute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
