[
https://issues.apache.org/jira/browse/FLINK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094573#comment-18094573
]
Martijn Visser commented on FLINK-33186:
----------------------------------------
Recurred on master as a {*}silent hang{*}, which is a different manifestation
from the {{CheckpointException: Task local checkpoint failure}} in the original
report.
* Run: GHA master, commit 0b16dd32ab1 (2026-07-07),
[https://github.com/apache/flink/actions/runs/28875060368]
* Method: {{testRestoreAfterSomeTasksFinished}}
* Symptom: no exception. The test thread parks on
{{miniCluster.requestJobResult(...).get()}}
({{{}CheckpointAfterAllTasksFinishedITCase.java:138{}}}) and never returns; the
surefire fork ran ~84 min until the CI watchdog dumped threads. In the dump the
restored job's {{passA -> Sink: sinkA}} subtasks are alive and parked in the
mailbox loop with end-of-input reached, waiting for a final checkpoint mail
that never arrives. The restored job uses checkpoint interval
{{{}Long.MAX_VALUE{}}}, so nothing retries.
This reproduced on a commit that already includes FLINK-39481 (#28608, merged
2026-07-06), so that finish-drain fix does not cover this case.
Working hypothesis (not yet confirmed, would appreciate a
checkpointing/scheduling committer's eyes): the final checkpoint is gated on
{{VertexEndOfDataListener.areAllTasksEndOfData()}} returning true, which
requires every vertex to reach {{{}cardinality == parallelism{}}}. If
{{FINISHED_ON_RESTORE}} subtasks do not re-report end-of-data after restore,
that condition is never met, {{SchedulerBase.notifyEndOfData}} never fires
{{{}triggerCheckpoint(CONFIGURED){}}}, and the finished-on-restore operators
wait forever. The intermittency likely comes from the savepoint being taken via
{{waitForSubtasksToFinish(..., false)}} (anyMatch), so it can capture a
partial-finish state where only some of the four subtasks have finished.
Thread dump available in the run above (test thread and the alive task threads).
CC [~pnowojski] [~roman]
> CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> fails on AZP
> -------------------------------------------------------------------------------------
>
> Key: FLINK-33186
> URL: https://issues.apache.org/jira/browse/FLINK-33186
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.19.0, 1.18.1, 2.4.0
> Reporter: Sergey Nuyanzin
> Assignee: Jiang Xin
> Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8762
> fails as
> {noformat}
> Sep 28 01:23:43 Caused by:
> org.apache.flink.runtime.checkpoint.CheckpointException: Task local
> checkpoint failure.
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2248)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2235)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:817)
> Sep 28 01:23:43 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Sep 28 01:23:43 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Sep 28 01:23:43 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
