[
https://issues.apache.org/jira/browse/FLINK-35335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120042#comment-18120042
]
Martijn Visser commented on FLINK-35335:
----------------------------------------
Two more on master, both nightlies on 2026-09-27. The region run never finished
and the legs were killed at the time budget:
https://github.com/apache/flink/actions/runs/36287526624/job/108532247145
(AdaptiveScheduler, JDK 17)
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=79476&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3
(test_cron_azure, JDK 17)
{code}
=== WARNING: This task took already 95% of the available time budget of 239
minutes ===
"ForkJoinPool-1-worker-1" #19 daemon prio=5 os_prio=0 cpu=2499.74ms
elapsed=12835.20s
java.lang.Thread.State: WAITING (parking)
at
java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2072)
at
org.apache.flink.test.util.TestUtils.submitJobAndWaitForResult(TestUtils.java:101)
at
org.apache.flink.test.checkpointing.StreamFaultToleranceTestBase.runCheckpointedProgram(StreamFaultToleranceTestBase.java:137)
"Source: Custom Source -> Filter (5/12)#173114" #41356575 prio=5 os_prio=0
cpu=1.42ms elapsed=0.01s
{code}
The GHA fork log has the job in an endless restart loop. It failed once in the
network stack after some Source -> Filter and Map -> Map subtasks had finished,
and every restore after that failed on empty state:
{code}
02:30:16,458 [Flat Map -> Sink: Unnamed (2/12)#0] WARN
org.apache.flink.runtime.taskmanager.Task [] - Flat Map -> Sink: Unnamed
(2/12)#0 switched from RUNNING to FAILED with failure cause:
java.lang.IllegalStateException
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:177)
at
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.convertToPriorityEvent(RemoteInputChannel.java:1078)
at
org.apache.flink.runtime.io.network.partition.consumer.IndexedInputGate.convertToPriorityEvent(IndexedInputGate.java:66)
at
org.apache.flink.streaming.runtime.io.checkpointing.AlternatingCollectingBarriersUnaligned.announcementReceived(AlternatingCollectingBarriersUnaligned.java:55)
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrierAnnouncement(SingleCheckpointBarrierHandler.java:318)
...
02:30:16,682 [Source: Custom Source -> Filter (12/12)#1] WARN
org.apache.flink.runtime.taskmanager.Task [] - Source: Custom Source -> Filter
(12/12)#1 switched from INITIALIZING to FAILED with failure cause:
java.lang.Exception: Failed to restore state to function: Test failed due to
unexpected recovered state size 0
Caused by: java.lang.RuntimeException: Test failed due to unexpected recovered
state size 0
at
org.apache.flink.test.checkpointing.StateCheckpointedITCase$StringRichFilterFunction.restoreState(StateCheckpointedITCase.java:241)
{code}
The Azure leg has no fork log; its dump shows the same loop at attempt #28043.
FLINK-31036 had the same restart loop on empty state.
> StateCheckpointedITCase failed fatally with 127 exit code
> ---------------------------------------------------------
>
> Key: FLINK-35335
> URL: https://issues.apache.org/jira/browse/FLINK-35335
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.19.1
> Reporter: Ryan Skraba
> Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59499&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=8379
> {code}
> May 13 01:50:22 01:50:22.272 [INFO] Tests run: 6, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 30.03 s -- in
> org.apache.flink.test.streaming.runtime.CacheITCase
> May 13 01:50:23 01:50:23.142 [INFO] Tests run: 1, Failures: 0, Errors: 0,
> Skipped: 0, Time elapsed: 5.234 s -- in
> org.apache.flink.test.streaming.experimental.CollectITCase
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.611 [INFO] Results:
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.611 [WARNING] Tests run: 1960, Failures: 0, Errors:
> 0, Skipped: 25
> May 13 01:50:23 01:50:23.611 [INFO]
> May 13 01:50:23 01:50:23.674 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.674 [INFO] BUILD FAILURE
> May 13 01:50:23 01:50:23.674 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.676 [INFO] Total time: 41:24 min
> May 13 01:50:23 01:50:23.677 [INFO] Finished at: 2024-05-13T01:50:23Z
> May 13 01:50:23 01:50:23.677 [INFO]
> ------------------------------------------------------------------------
> May 13 01:50:23 01:50:23.677 [WARNING] The requested profile
> "skip-webui-build" could not be activated because it does not exist.
> May 13 01:50:23 01:50:23.678 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.2.2:test (integration-tests)
> on project flink-tests:
> May 13 01:50:23 01:50:23.678 [ERROR]
> May 13 01:50:23 01:50:23.678 [ERROR] Please refer to
> /__w/2/s/flink-tests/target/surefire-reports for the individual test results.
> May 13 01:50:23 01:50:23.678 [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> May 13 01:50:23 01:50:23.678 [ERROR] ExecutionException The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> May 13 01:50:23 01:50:23.678 [ERROR] Command was /bin/sh -c cd
> '/__w/2/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/2/s/flink-tests/target/surefire/surefirebooter-20240513010926195_686.jar'
> '/__w/2/s/flink-tests/target/surefire' '2024-05-13T01-09-20_665-jvmRun1'
> 'surefire-20240513010926195_684tmp' 'surefire_206-20240513010926195_685tmp'
> May 13 01:50:23 01:50:23.679 [ERROR] Error occurred in starting fork, check
> output in log
> May 13 01:50:23 01:50:23.679 [ERROR] Process Exit Code: 127
> May 13 01:50:23 01:50:23.679 [ERROR] Crashed tests:
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.flink.test.checkpointing.StateCheckpointedITCase
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> May 13 01:50:23 01:50:23.679 [ERROR] Command was /bin/sh -c cd
> '/__w/2/s/flink-tests' && '/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/2/s/flink-tests/target/surefire/surefirebooter-20240513010926195_686.jar'
> '/__w/2/s/flink-tests/target/surefire' '2024-05-13T01-09-20_665-jvmRun1'
> 'surefire-20240513010926195_684tmp' 'surefire_206-20240513010926195_685tmp'
> May 13 01:50:23 01:50:23.679 [ERROR] Error occurred in starting fork, check
> output in log
> May 13 01:50:23 01:50:23.679 [ERROR] Process Exit Code: 127
> May 13 01:50:23 01:50:23.679 [ERROR] Crashed tests:
> May 13 01:50:23 01:50:23.679 [ERROR]
> org.apache.flink.test.checkpointing.StateCheckpointedITCase
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:456)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:418)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
> May 13 01:50:23 01:50:23.679 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
> {code}
> In the maven logs, {{runCheckpointedProgram[FailoverStrategy:
> RestartPipelinedRegionFailoverStrategy]}} is started but never completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
