[
https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13673947#comment-13673947
]
Bruce Nelson commented on HIVE-4650:
------------------------------------
If "hive.auto.convert.join = false" is set then the all the query stages work
OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin
working.
> Getting Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after
> upgrade to Hive-0.11.0.x from hive-0.10.0.x
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-4650
> URL: https://issues.apache.org/jira/browse/HIVE-4650
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.11.0
> Environment: HortonWorks 1.3 distro on x86_64 Centos 6
> Reporter: Bruce Nelson
>
> working from a simple table in Hive
> hive> desc cmnt
> > ;
> OK
> x1 int None
> x2 int None
> x3 int None
> x4 int None
> y double None
> hive> select * from cmnt;
> OK
> 7 26 6 60 78.5
> 1 29 15 52 74.3
> 11 56 8 20 104.3
> 11 31 8 47 87.6
> 7 52 6 33 95.9
> 11 55 9 22 109.2
> 3 71 17 6 102.7
> 1 31 22 44 72.5
> 2 54 18 22 93.1
> 21 47 4 26 115.9
> 1 40 23 34 83.8
> 11 66 9 12 113.3
> 10 68 8 12 109.4
> A query that joins and transforms against this table :
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from (
> select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) *
> 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4)
> VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1
> VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join
> (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003)
> mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001)
> sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004)
> sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2
> VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7)
> obj1_6 ;
> Generates during Stage-3 :
> setting HADOOP_USER_NAME test
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55 Starting to launch local task to process map join;
> maximum memory = 1065484288
> 2013-06-03 12:40:56 Processing rows: 1 Hashtable size: 1
> Memory usage: 7175528 rate: 0.007
> 2013-06-03 12:40:56 Dump the hashtable into file:
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56 Upload 1 File to:
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> File size: 334
> 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL =
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill
> job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%, reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%, reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_000002 (and more) from job
> job_201306022123_0045
> Task with the most failures(4):
> -----
> Task ID:
> task_201306022123_0045_m_000000
> URL:
>
> http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
> ... 14 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> If "hive.auto.convert.join = false" is set then the all the query stages work
> OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin
> working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
