[
https://issues.apache.org/jira/browse/HIVE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris McConnell updated HIVE-3697:
----------------------------------
Affects Version/s: 0.9.0
> External JAR files on HDFS can lead to race condition with
> hive.downloaded.resources.dir
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-3697
> URL: https://issues.apache.org/jira/browse/HIVE-3697
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Chris McConnell
>
> I've seen situations where utilizing JAR files on HDFS can cause job failures
> via CNFE or JVM crashes.
> This is difficult to replicate, seems to be related to JAR size, latency
> between client and HDFS cluster, but I've got some example stack traces
> below. Seems that the calls made to FileSystem (copyToLocal) which are static
> and will be executed to delete the current local copy can cause the file(s)
> to be removed during job processing.
> We should consider changing the default for hive.downloaded.resources.dir to
> include some level of uniqueness per job. We should not consider
> hive.session.id however, as execution of multiple statements via the same
> user/session which might access the same JAR files will utilize the same
> session.
> A proposal might be to utilize System.nanoTime() -- which might be enough to
> avoid the issue, although it's not perfect (depends on JVM and system for
> level of precision) as part of the default
> (/tmp/${user.name}/resources/System.nanoTime()/).
> If anyone else has hit this, would like to capture environment information as
> well. Perhaps there is something else at play here.
> Here are some examples of the errors:
> for i in {0..2}; do hive -S -f query.q& done
> [2] 48405
> [3] 48406
> [4] 48407
> % #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0x7) at pc=0x00007fb10bd931f0, pid=48407, tid=140398456698624
> #
> # JRE version: 6.0_31-b04
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [libzip.so+0xb1f0] __int128+0x60
> #
> # An error report file with more information is saved as:
> # /home/.../hs_err_pid48407.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> java.lang.NoClassDefFoundError: com/example/udf/Lower
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:105)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:75)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:63)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:439)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:449)
> at
> org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:485)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:692)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.ClassNotFoundException: com.example.udf.Lower
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 24 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.FunctionTask
> Another:
> for i in {0..2}; do hive -S -f query.q& done
> [1] 16294
> [2] 16295
> [3] 16296
> []$ Couldn't create directory /tmp/ctm/resources/
> Couldn't create directory /tmp/ctm/resources/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
