[
https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-4650:
---------------------------------
Attachment: join32_lessSize.q.out
Bruce: I meant the patch HIVE-4502.D10695.patch is the one that fixes the
issue. HIVE-4502-1.patch is not going to be the patch for it as it does not fix
this case. Those differences that you mention is because of the inherent
version differences and will need to be resolved. I have attached the results
from a test I ran to verify a case similar to yours.
> Getting Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after
> upgrade to Hive-0.11.0.x from hive-0.10.0.x
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-4650
> URL: https://issues.apache.org/jira/browse/HIVE-4650
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.11.0
> Environment: HortonWorks 1.3 distro on x86_64 Centos 6
> Reporter: Bruce Nelson
> Attachments: join32_lessSize.q.out
>
>
> working from a simple table in Hive
> hive> desc cmnt
> > ;
> OK
> x1 int None
> x2 int None
> x3 int None
> x4 int None
> y double None
> hive> select * from cmnt;
> OK
> 7 26 6 60 78.5
> 1 29 15 52 74.3
> 11 56 8 20 104.3
> 11 31 8 47 87.6
> 7 52 6 33 95.9
> 11 55 9 22 109.2
> 3 71 17 6 102.7
> 1 31 22 44 72.5
> 2 54 18 22 93.1
> 21 47 4 26 115.9
> 1 40 23 34 83.8
> 11 66 9 12 113.3
> 10 68 8 12 109.4
> A query that joins and transforms against this table :
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from (
> select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) *
> 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4)
> VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1
> VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join
> (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003)
> mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001)
> sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004)
> sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2
> VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7)
> obj1_6 ;
> Generates during Stage-3 :
> setting HADOOP_USER_NAME test
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55 Starting to launch local task to process map join;
> maximum memory = 1065484288
> 2013-06-03 12:40:56 Processing rows: 1 Hashtable size: 1
> Memory usage: 7175528 rate: 0.007
> 2013-06-03 12:40:56 Dump the hashtable into file:
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56 Upload 1 File to:
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> File size: 334
> 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL =
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill
> job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%, reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%, reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_000002 (and more) from job
> job_201306022123_0045
> Task with the most failures(4):
> -----
> Task ID:
> task_201306022123_0045_m_000000
> URL:
>
> http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
> ... 14 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> If "hive.auto.convert.join = false" is set then the all the query stages work
> OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin
> working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
