[
https://issues.apache.org/jira/browse/HIVE-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733660#comment-13733660
]
Abin Shahab commented on HIVE-5016:
-----------------------------------
Root cause of this issue is Not picking the YarnRunner as the job runner when
hive.exec.mode.local.auto=true
mapreduce.framework.name gets set to 'local' instead of 'yarn'. This results in
the LocalJobRunner being used as the JobRunner, and messes up the path
creation.
> Local mode FileNotFoundException: File does not exist
> -----------------------------------------------------
>
> Key: HIVE-5016
> URL: https://issues.apache.org/jira/browse/HIVE-5016
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.10.0
> Environment: Centos 6.3 (final)
> Hadoop 2.0.2-alpha
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Hive libs:
> ls -1 lib/
> antlr-2.7.7.jar
> antlr-runtime-3.0.1.jar
> avro-1.7.1.jar
> avro-mapred-1.7.1.jar
> commons-cli-1.2.jar
> commons-codec-1.4.jar
> commons-collections-3.2.1.jar
> commons-compress-1.4.1.jar
> commons-configuration-1.6.jar
> commons-dbcp-1.4.jar
> commons-lang-2.4.jar
> commons-logging-1.0.4.jar
> commons-logging-api-1.0.4.jar
> commons-pool-1.5.4.jar
> datanucleus-connectionpool-2.0.3.jar
> datanucleus-core-2.0.3.jar
> datanucleus-enhancer-2.0.3.jar
> datanucleus-rdbms-2.0.3.jar
> derby-10.4.2.0.jar
> guava-r09.jar
> hbase-0.92.0.jar
> hbase-0.92.0-tests.jar
> hive-builtins-0.10.0.jar
> hive-cli-0.10.0.jar
> hive-common-0.10.0.jar
> hive-contrib-0.10.0.jar
> hive-exec-0.10.0.jar
> hive-hbase-handler-0.10.0.jar
> hive-hwi-0.10.0.jar
> hive-hwi-0.10.0.war
> hive-jdbc-0.10.0.jar
> hive-metastore-0.10.0.jar
> hive-pdk-0.10.0.jar
> hive-serde-0.10.0.jar
> hive-service-0.10.0.jar
> hive-shims-0.10.0.jar
> jackson-core-asl-1.8.8.jar
> jackson-jaxrs-1.8.8.jar
> jackson-mapper-asl-1.8.8.jar
> jackson-xc-1.8.8.jar
> JavaEWAH-0.3.2.jar
> javolution-5.5.1.jar
> jdo2-api-2.3-ec.jar
> jetty-6.1.26.jar
> jetty-util-6.1.26.jar
> jline-0.9.94.jar
> json-20090211.jar
> libfb303-0.9.0.jar
> libthrift-0.9.0.jar
> log4j-1.2.16.jar
> php
> py
> servlet-api-2.5-20081211.jar
> slf4j-api-1.6.1.jar
> slf4j-log4j12-1.6.1.jar
> sqlline-1_0_2.jar
> stringtemplate-3.1-b1.jar
> xz-1.0.jar
> zookeeper-3.4.3.jar
> Reporter: Abin Shahab
> Priority: Critical
>
> Hive jobs in local mode fail with the error posted below. The jar file that's
> not being found exists and has the following access:
> > ls -l hive-0.10.0/lib/hive-builtins-0.10.0.jar
> rw-rw-r-- 1 ashahab ashahab 3914 Dec 18 2012
> hive-0.10.0/lib/hive-builtins-0.10.0.jar
> Steps to reproduce:
> hive> set hive.exec.mode.local.auto=true;
> hive> set hive.exec.mode.local.auto;
> hive.exec.mode.local.auto=true
> hive> select count(*) from abin_test_table;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> 13/08/06 21:37:11 WARN conf.Configuration:
> file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 13/08/06 21:37:11 WARN conf.Configuration:
> file:/tmp/ashahab/hive_2013-08-06_21-37-09_046_3263640403676309186/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Execution log at:
> /tmp/ashahab/ashahab_20130806213737_7d26b796-5f55-44ca-a755-8898153d963b.log
> java.io.FileNotFoundException: File does not exist:
> /home/ashahab/dev/hive-0.10.0/lib/hive-builtins-0.10.0.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:252)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:290)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:361)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:689)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
