also another important question: is it possible to read real hive tables on HDFS even though you are running hive in local mode? in most cases the tables are complex and it's a chore to replicate them to local and set up your derby local metastore
in the case of PIG, copying these input files to local is less of a chore since PIG does not need the metastore On Wed, Jun 18, 2014 at 11:14 AM, Yang <teddyyyy...@gmail.com> wrote: > I tried to run hive in local mode to debug a script (possibly with UDF) so > that I could attach it to eclipse for debugging. > > I followed the advice of > https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Hive,Map-ReduceandLocal-Mode > > i.e. > > hive> SET mapred.local.dir = /tmp/yyang15/mapred/local; > hive> SET mapred.job.tracker=local; > hive> SET hive.exec.mode.local.auto=true; > hive> select * from ( select * from yy union all select * from zz ) kk; > Automatically selecting local only mode for query > Query ID = yyang15_20140618111212_742a8332-cec9-413d-8536-1fcb2e743f85 > Total jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > 14/06/18 11:12:29 WARN conf.Configuration: > file:/tmp/yyang15/hive_2014-06-18_11-12-23_978_2698605745140935029-1/-local-10003/jobconf.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 14/06/18 11:12:29 WARN conf.Configuration: > file:/tmp/yyang15/hive_2014-06-18_11-12-23_978_2698605745140935029-1/-local-10003/jobconf.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > Execution log at: > /tmp/yyang15/yyang15_20140618111212_742a8332-cec9-413d-8536-1fcb2e743f85.log > java.io.FileNotFoundException: File does not exist: > hdfs://manny-lvs/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.2.0-402.jar > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) > at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420) > at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:741) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Job Submission failed with exception 'java.io.FileNotFoundException(File > does not exist: > hdfs://manny-lvs/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.2.0-402.jar)' > Execution failed with exit status: 1 > Obtaining error information > > Task failed! > Task ID: > Stage-1 > > Logs: > > /tmp/yyang15/hive.log > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > > > > > it seems that hive is trying to access a local jar but assuming the base > dir to be HDFS. did anybody get this to work? > > > Thanks > Yang >
