Hi Yin: Thanks for the patch, I patch it and pass this testcase, I will use it with our hive11 production test.
在 2013年8月1日星期四，上午5:09，Yin Huai 写道： > I just uploaded a patch to https://issues.apache.org/jira/browse/HIVE-4968. > You can try it and see if the problem has been resolved for your query. > > > On Wed, Jul 31, 2013 at 11:21 AM, Yin Huai <huaiyin....@gmail.com > (mailto:huaiyin....@gmail.com)> wrote: > > Seems it is another problem. > > Can you try > > > > > > SELECT * > > FROM (SELECT VAL001 x1, > > VAL002 x2, > > VAL003 x3, > > VAL004 x4, > > VAL005 y > > FROM (SELECT /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001, > > (VAL002- mu2) * 1/(sd2) VAL002, > > (VAL003- mu3) * 1/(sd3) VAL003, > > (VAL004- mu4) * 1/(sd4) VAL004, > > (VAL005- mu5) * 1/(sd5) VAL005 > > FROM (SELECT x1 VAL001, > > > > x2 VAL002, > > x3 VAL003, > > x4 VAL004, > > y VAL005 > > FROM cmnt) v3 > > > > JOIN (SELECT count(*) c, > > avg(VAL001) mu1, > > avg(VAL002) mu2, > > avg(VAL003) mu3, > > avg(VAL004) mu4, > > avg(VAL005) mu5, > > stddev_pop(VAL001) sd1, > > stddev_pop(VAL002) sd2, > > stddev_pop(VAL003) sd3, > > stddev_pop(VAL004) sd4, > > stddev_pop(VAL005) sd5 > > FROM (SELECT * > > FROM (SELECT x1 VAL001, > > x2 VAL002, > > x3 VAL003, > > x4 VAL004, > > y VAL005 > > FROM cmnt) obj1_3) v1) v2) obj1_7) obj1_6; > > > > Also, cmnt in v3 will be used to create the hash table. Seems the part of > > code in converting Join to MapJoin does not play well with this part of > > your original query > > > > > > SELECT * > > FROM > > (SELECT x1 VAL001, > > x2 VAL002, > > x3 VAL003, > > x4 VAL004, > > y VAL005 > > FROM cmnt) obj1_3) v3 > > > > > > I have created https://issues.apache.org/jira/browse/HIVE-4968 to address > > this issue. > > > > > > > > > > On Sun, Jul 28, 2013 at 11:46 PM, <wzc1...@gmail.com > > (mailto:wzc1...@gmail.com)> wrote: > > > Hi: > > > I attach the output of EXPLAIN, and the hive I use is compiled from trunk > > > and my hadoop version is 1.0.1. I use default hive configuration. > > > > > > > > > -- > > > wzc1...@gmail.com (mailto:wzc1...@gmail.com) > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > 在 2013年7月29日星期一，下午1:08，Yin Huai 写道： > > > > > > > Hi, > > > > > > > > Can you also post the output of EXPLAIN? The execution plan may be > > > > helpful to locate the problem. > > > > > > > > Thanks, > > > > > > > > Yin > > > > > > > > > > > > On Sun, Jul 28, 2013 at 8:06 PM, <wzc1...@gmail.com > > > > (mailto:wzc1...@gmail.com)> wrote: > > > > > What I mean by "not pass the testcase in HIVE-4650" is that I compile > > > > > the trunk code and run the query in HIVE-4650: > > > > > SELECT * > > > > > FROM > > > > > (SELECT VAL001 x1, > > > > > VAL002 x2, > > > > > VAL003 x3, > > > > > VAL004 x4, > > > > > VAL005 y > > > > > FROM > > > > > (SELECT /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) > > > > > VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) > > > > > VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 > > > > > FROM > > > > > (SELECT * > > > > > FROM > > > > > (SELECT x1 VAL001, > > > > > x2 VAL002, > > > > > x3 VAL003, > > > > > x4 VAL004, > > > > > y VAL005 > > > > > FROM cmnt) obj1_3) v3 > > > > > JOIN > > > > > (SELECT count(*) c, > > > > > avg(VAL001) mu1, > > > > > avg(VAL002) mu2, > > > > > avg(VAL003) mu3, > > > > > avg(VAL004) mu4, > > > > > avg(VAL005) mu5, > > > > > stddev_pop(VAL001) sd1, > > > > > stddev_pop(VAL002) sd2, > > > > > stddev_pop(VAL003) sd3, > > > > > stddev_pop(VAL004) sd4, > > > > > stddev_pop(VAL005) sd5 > > > > > FROM > > > > > (SELECT * > > > > > FROM > > > > > (SELECT x1 VAL001, > > > > > x2 VAL002, > > > > > x3 VAL003, > > > > > x4 VAL004, > > > > > y VAL005 > > > > > FROM cmnt) obj1_3) v1) v2) obj1_7) obj1_6 ; > > > > > > > > > > > > > > > and it still fail at the same place: > > > > > … > > > > > Diagnostic Messages for this Task: > > > > > java.lang.RuntimeException: > > > > > org.apache.hadoop.hive.ql.metadata.HiveException: > > > > > java.lang.NullPointerException > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162) > > > > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > > > > > at > > > > > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > > > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > > > > > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > > at javax.security.auth.Subject.doAs(Subject.java:416) > > > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) > > > > > at org.apache.hadoop.mapred.Child.main(Child.java:249) > > > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > > > > > java.lang.NullPointerException > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611) > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) > > > > > ... 8 more > > > > > Caused by: java.lang.NullPointerException > > > > > at > > > > > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186) > > > > > ... 14 more > > > > > > > > > > > > > > > -- > > > > > wzc1...@gmail.com (mailto:wzc1...@gmail.com) > > > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > > > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > > > > > 在 2013年7月28日星期日，下午8:08，wzc1...@gmail.com (mailto:wzc1...@gmail.com) > > > > > 写道： > > > > > > > > > > > hi all: > > > > > > > > > > > > We are currently testing hive 0.11 against our production > > > > > > environment and run into some problems. Some of them are related to > > > > > > the param "hive.auto.convert.join". > > > > > > We disable this param and some failed testcases passed. By > > > > > > searching in hive jira issues I find that the patch in > > > > > > HIVE-4650(https://issues.apache.org/jira/browse/HIVE-4650) may be > > > > > > helpful. > > > > > > I compile the newest code in trunk and try the failed testcase in > > > > > > HIVE-4650, but it doesn't pass. It seems that this issue is not > > > > > > fixed while it's closed. > > > > > > > > > > > > Am I missed something? > > > > > > > > > > > > -- > > > > > > wzc1...@gmail.com (mailto:wzc1...@gmail.com) > > > > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > > > > > > > 已使用 Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > > > > > > > > > >
