[
https://issues.apache.org/jira/browse/FLINK-21929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang resolved FLINK-21929.
------------------------------
Resolution: Fixed
> flink-statebackend-rocksdb crashes with Error occurred in starting fork
> -----------------------------------------------------------------------
>
> Key: FLINK-21929
> URL: https://issues.apache.org/jira/browse/FLINK-21929
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.13.0
> Reporter: Robert Metzger
> Assignee: Yun Tang
> Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
> Attachments: image-2021-03-23-13-18-41-836.png
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=9001&view=results
> {code}
> 2021-03-23T09:11:12.1861967Z [INFO] BUILD FAILURE
> 2021-03-23T09:11:12.1863007Z [INFO]
> ------------------------------------------------------------------------
> 2021-03-23T09:11:12.1863492Z [INFO] Total time: 42:35 min
> 2021-03-23T09:11:12.1864171Z [INFO] Finished at: 2021-03-23T09:11:12+00:00
> 2021-03-23T09:11:12.8003245Z [INFO] Final Memory: 137M/806M
> 2021-03-23T09:11:12.8006310Z [INFO]
> ------------------------------------------------------------------------
> 2021-03-23T09:11:12.8082409Z [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on
> project flink-statebackend-rocksdb_2.11: There are test failures.
> 2021-03-23T09:11:12.8086652Z [ERROR]
> 2021-03-23T09:11:12.8092462Z [ERROR] Please refer to
> /__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports
> for the individual test results.
> 2021-03-23T09:11:12.8096948Z [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> 2021-03-23T09:11:12.8101388Z [ERROR] ExecutionException Error occurred in
> starting fork, check output in log
> 2021-03-23T09:11:12.8105868Z [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException Error occurred in starting fork, check output in log
> 2021-03-23T09:11:12.8110518Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> 2021-03-23T09:11:12.8115518Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:382)
> 2021-03-23T09:11:12.8120811Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:297)
> 2021-03-23T09:11:12.8126356Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> 2021-03-23T09:11:12.8127129Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> 2021-03-23T09:11:12.8131291Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> 2021-03-23T09:11:12.8132369Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> 2021-03-23T09:11:12.8133397Z [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> 2021-03-23T09:11:12.8134116Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> 2021-03-23T09:11:12.8134793Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> 2021-03-23T09:11:12.8135621Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> 2021-03-23T09:11:12.8136323Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> 2021-03-23T09:11:12.8141570Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> 2021-03-23T09:11:12.8142374Z [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> 2021-03-23T09:11:12.8145665Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> 2021-03-23T09:11:12.8146407Z [ERROR] at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> 2021-03-23T09:11:12.8148835Z [ERROR] at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> 2021-03-23T09:11:12.8151299Z [ERROR] at
> org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> 2021-03-23T09:11:12.8152244Z [ERROR] at
> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> 2021-03-23T09:11:12.8152806Z [ERROR] at
> org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> 2021-03-23T09:11:12.8155818Z [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-03-23T09:11:12.8159757Z [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-03-23T09:11:12.8177288Z [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-03-23T09:11:12.8178021Z [ERROR] at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-03-23T09:11:12.8179802Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> 2021-03-23T09:11:12.8183929Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> 2021-03-23T09:11:12.8187563Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> 2021-03-23T09:11:12.8192413Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> 2021-03-23T09:11:12.8196538Z [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred
> in starting fork, check output in log
> 2021-03-23T09:11:12.8201660Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:622)
> 2021-03-23T09:11:12.8203999Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
> 2021-03-23T09:11:12.8204879Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:371)
> 2021-03-23T09:11:12.8205665Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:347)
> 2021-03-23T09:11:12.8206513Z [ERROR] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-03-23T09:11:12.8207169Z [ERROR] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-03-23T09:11:12.8209376Z [ERROR] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-03-23T09:11:12.8209955Z [ERROR] at java.lang.Thread.run(Thread.java:748)
> 2021-03-23T09:11:12.8211107Z [ERROR] -> [Help 1]
> 2021-03-23T09:11:12.8211559Z [ERROR]
> 2021-03-23T09:11:12.8309016Z [ERROR] To see the full stack trace of the
> errors, re-run Maven with the -e switch.
> 2021-03-23T09:11:12.8310211Z [ERROR] Re-run Maven using the -X switch to
> enable full debug logging.
> 2021-03-23T09:11:12.8311401Z [ERROR]
> 2021-03-23T09:11:12.8311976Z [ERROR] For more information about the errors
> and possible solutions, please read the following articles:
> 2021-03-23T09:11:12.8312635Z [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> 2021-03-23T09:11:12.8313300Z [ERROR]
> 2021-03-23T09:11:12.8314248Z [ERROR] After correcting the problems, you can
> resume the build with the command
> 2021-03-23T09:11:12.8315462Z [ERROR] mvn <goals> -rf
> :flink-statebackend-rocksdb_2.11
> 2021-03-23T09:11:12.8687280Z Process exited with EXIT CODE: 1.
> 2021-03-23T09:11:12.8687830Z Trying to KILL watchdog (359).
> 2021-03-23T09:11:12.8696302Z /__w/1/s/tools/ci/watchdog.sh: line 100: 359
> Terminated watchdog
> 2021-03-23T09:11:16.4147696Z Searching for .dump, .dumpstream and related
> files in '/__w/1/s'
> 2021-03-23T09:11:21.6569652Z Moving
> '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6645588Z Moving
> '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6683479Z Moving
> '/__w/1/s/flink-runtime/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6726888Z Moving
> '/__w/1/s/flink-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6772031Z Moving
> '/__w/1/s/flink-metrics/flink-metrics-core/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6807831Z Moving
> '/__w/1/s/flink-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6847619Z Moving
> '/__w/1/s/flink-runtime-web/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:21.6882147Z Moving
> '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/core.18126' to
> target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1251457Z Moving
> '/__w/1/s/flink-state-backends/flink-statebackend-rocksdb/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1287286Z Moving
> '/__w/1/s/flink-test-utils-parent/flink-test-utils/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1322850Z Moving
> '/__w/1/s/flink-clients/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1360757Z Moving
> '/__w/1/s/flink-streaming-java/java_pid5009.hprof' to target directory
> ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1421701Z Moving
> '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun2.dump'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1460325Z Moving
> '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1492941Z Moving
> '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805-jvmRun1.dump'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1523043Z Moving
> '/__w/1/s/flink-streaming-java/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1554119Z Moving
> '/__w/1/s/flink-optimizer/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1589618Z Moving
> '/__w/1/s/flink-scala/target/surefire-reports/2021-03-23T08-28-44_805.dumpstream'
> to target directory ('/__w/_temp/debug_files/')
> 2021-03-23T09:11:22.1638143Z Compressing debug files
> {code}
> I'm not sure if this includes one or two issues.
> One issue seems to be in the flink-streaming-java tests:
> {code}
> # Created at 2021-03-23T09:03:49.666
> Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream
> 'java.lang.OutOfMemoryError: Java heap space'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'java.lang.OutOfMemoryError: Java heap
> space'.
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
> at java.lang.Thread.run(Thread.java:748)
> # Created at 2021-03-23T09:03:49.667
> Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream
> 'Dumping heap to java_pid5009.hprof ...'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Dumping heap to java_pid5009.hprof ...'.
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
> at java.lang.Thread.run(Thread.java:748)
> # Created at 2021-03-23T09:03:49.693
> Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream
> 'Heap dump file created [2834354 bytes in 0.027 secs]'.
> java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma
> after third character in command 'Heap dump file created [2834354 bytes in
> 0.027 secs]'.
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
> at
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> There's a coredump included:
> !image-2021-03-23-13-18-41-836.png!
> Looks like the testInitialSizeCompoutation test is causing the / a failure.
> The second issue is (which is reported as a test failure):
> {code}
> # Created at 2021-03-23T09:09:39.376
> Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> # Created at 2021-03-23T09:09:39.535
> Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
> # Created at 2021-03-23T09:11:10.037
> pure virtual method called
> # Created at 2021-03-23T09:11:10.037
> terminate called without an active exception
> # Created at 2021-03-23T09:11:12.170
> Aborted (core dumped)
> {code}
> From the coredump, I see the following:
> {code}
> (gdb) where
> #0 0x00007f9343508438 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x00007f934350a03a in __GI_abort () at abort.c:89
> #2 0x00007f9341b8e84d in __gnu_cxx::__verbose_terminate_handler() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #3 0x00007f9341b8c6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #4 0x00007f9341b8c701 in std::terminate() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #5 0x00007f9341b8d23f in __cxa_pure_virtual () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #6 0x00007f92ed39c0d5 in rocksdb::DBImpl::CloseHelper() () from
> /tmp/librocksdbjni8334465068904677424.so
> #7 0x00007f92ed3a717b in rocksdb::DBImpl::~DBImpl() () from
> /tmp/librocksdbjni8334465068904677424.so
> #8 0x00007f92ed3a7451 in rocksdb::DBImpl::~DBImpl() () from
> /tmp/librocksdbjni8334465068904677424.so
> #9 0x00007f932d801aa8 in ?? ()
> #10 0x0000000085677fe8 in ?? ()
> #11 0x00007f931c8da910 in ?? ()
> #12 0x00007f931c8da968 in ?? ()
> #13 0x00007f932d007ffd in ?? ()
> #14 0x0000000000000000 in ?? ()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
