Marton Elek created HDDS-2308:
---------------------------------
Summary: Switch to centos with the apache/ozone-build docker image
Key: HDDS-2308
URL: https://issues.apache.org/jira/browse/HDDS-2308
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek
Attachments: hs_err_pid16346.log
I realized multiple JVM crashes in the daily builds:
{code:java}
ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7569723928289175829tmp surefire_947955725320624341206tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
[ERROR] ExecutionException The forked VM terminated without properly
saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7227403571189445391tmp surefire_1011197392458143645283tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire3938612864214747736tmp surefire_933162535733309260236tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] ExecutionException The forked VM terminated without properly
saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7569723928289175829tmp surefire_947955725320624341206tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139 {code}
Based on the crash log (uploaded) it's related to the rocksdb JNI interface.
In the current ozone-build docker image (which provides the environment for
build) we use alpine where musl libc is used instead of the main glibc. I think
it would be more safe to use the same glibc what is used in production.
I tested with centos based docker image and it seems to be more stable. Didn't
see any more JVM crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
