[
https://issues.apache.org/jira/browse/HIVE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495261#comment-13495261
]
Chris McConnell commented on HIVE-3697:
---------------------------------------
Hey Joe,
Thanks for the work around, seems to work well for me also. I'll look into the
possibility of changing the parent directory as well within the HiveConf to do
something similar.
Cheers,
Chris
> External JAR files on HDFS can lead to race condition with
> hive.downloaded.resources.dir
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-3697
> URL: https://issues.apache.org/jira/browse/HIVE-3697
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Chris McConnell
>
> I've seen situations where utilizing JAR files on HDFS can cause job failures
> via CNFE or JVM crashes.
> This is difficult to replicate, seems to be related to JAR size, latency
> between client and HDFS cluster, but I've got some example stack traces
> below. Seems that the calls made to FileSystem (copyToLocal) which are static
> and will be executed to delete the current local copy can cause the file(s)
> to be removed during job processing.
> We should consider changing the default for hive.downloaded.resources.dir to
> include some level of uniqueness per job. We should not consider
> hive.session.id however, as execution of multiple statements via the same
> user/session which might access the same JAR files will utilize the same
> session.
> A proposal might be to utilize System.nanoTime() -- which might be enough to
> avoid the issue, although it's not perfect (depends on JVM and system for
> level of precision) as part of the default
> (/tmp/${user.name}/resources/System.nanoTime()/).
> If anyone else has hit this, would like to capture environment information as
> well. Perhaps there is something else at play here.
> Here are some examples of the errors:
> for i in {0..2}; do hive -S -f query.q& done
> [2] 48405
> [3] 48406
> [4] 48407
> % #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0x7) at pc=0x00007fb10bd931f0, pid=48407, tid=140398456698624
> #
> # JRE version: 6.0_31-b04
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [libzip.so+0xb1f0] __int128+0x60
> #
> # An error report file with more information is saved as:
> # /home/.../hs_err_pid48407.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> java.lang.NoClassDefFoundError: com/example/udf/Lower
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:105)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:75)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:63)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:439)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:449)
> at
> org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:485)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:692)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.ClassNotFoundException: com.example.udf.Lower
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 24 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.FunctionTask
> Another:
> for i in {0..2}; do hive -S -f query.q& done
> [1] 16294
> [2] 16295
> [3] 16296
> []$ Couldn't create directory /tmp/ctm/resources/
> Couldn't create directory /tmp/ctm/resources/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
