Arnaud Linz created HIVE-24507:
----------------------------------
Summary: "File file:XXX.jar does not exist" when changing content
of "hive.reloadable.aux.jars.path" directory content
Key: HIVE-24507
URL: https://issues.apache.org/jira/browse/HIVE-24507
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 2.1.1
Reporter: Arnaud Linz
Purpose of hive.reloadable.aux.jars.path, introduced by
https://issues.apache.org/jira/browse/HIVE-7553 was do avoid scheduling
maintenance window for every jar change, but it is not enough.
On a large system, the lack of atomicity between the directory listing of jars
contained in hive.reloadable.aux.jars.path and the actual use of the file when
uploaded to the job's yarn resources lead to query failures, even if no jar/UDF
is used in the failing query (because it is a global parameter).
Stack trace sample:
{code:java}
File file:/XXX.jar does not exist at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:378) at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at
org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:703)
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:315)
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:207)
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:135)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at
org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576) at
org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562) at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:444) at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151) at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200) at
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334) at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
at
org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
It's probably not possible to achieve atomicity, but this lack of atomicity
should be taken into account and this error should be a warning. Actually, if a
jar is removed, it's probably because no query are using it any longer. And if
it was really used, it will trigger another ClassNotFound error later that,
with the warning log, can suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
