[
https://issues.apache.org/jira/browse/HIVE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670603#comment-13670603
]
Owen O'Malley commented on HIVE-4436:
-------------------------------------
+1, looks good.
> hive.exec.parallel=true doesn't work on hadoop-2
> ------------------------------------------------
>
> Key: HIVE-4436
> URL: https://issues.apache.org/jira/browse/HIVE-4436
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.9.0, 0.12.0
> Environment: Ubuntu LXC (hive-trunk), CDH 4 on Debian
> Reporter: Gopal V
> Assignee: Gopal V
> Attachments: HIVE-4436.patch, HIVE-4436-test.tgz, parallel_sorted.q
>
>
> While running a hive query with multiple independent stages,
> hive.exec.parallel is a valid optimization to use.
> The query tested has 3 MR jobs - the first job is the root dependency and the
> 2 further job depend on the first one.
> When hive.exec.parallel is turned on, the job fails with the following
> exception
> {code}
> java.io.IOException: java.lang.InterruptedException
> at org.apache.hadoop.ipc.Client.call(Client.java:1214)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy12.mkdirs(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> at $Proxy12.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:447)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2165)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2136)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:544)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1916)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:222)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:444)
> at
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:921)
> at org.apache.hadoop.ipc.Client.call(Client.java:1208)
> {code}
> The query plan is as follows
> {code}
> Stage-9 is a root stage
> Stage-8 depends on stages: Stage-9
> Stage-3 depends on stages: Stage-8
> Stage-0 depends on stages: Stage-3
> Stage-4 depends on stages: Stage-0
> Stage-5 depends on stages: Stage-8
> Stage-1 depends on stages: Stage-5
> Stage-6 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-9
> Map Reduce Local Work
> Stage: Stage-8
> Map Reduce
> Map Join Operator
> Stage: Stage-3
> Map Reduce
> Stage: Stage-0
> Move Operator
> Stage: Stage-4
> Stats-Aggr Operator
> Stage: Stage-5
> Map Reduce
> Stage: Stage-1
> Move Operator
> Stage: Stage-6
> Stats-Aggr Operator
> {code}
> -I cannot conclude that this is purely a hive issue, will file a bug on HDFS
> if that does show up during triage.-
> *Triaged* - set hive.stats.autogather=false; removes the bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
