[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714762#comment-13714762
]
Yin Huai commented on HIVE-4502:
--------------------------------
The failed query in correlationoptimizer3.q is ...
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT d.key AS key, d.cnt AS cnt, b.value as value
FROM (SELECT x.key as key, x.value as value FROM src1 x JOIN src y ON (x.key =
y.key)) b
JOIN (SELECT x.key as key, count(1) AS cnt FROM src1 x JOIN src y ON (x.key =
y.key) group by x.key) d
ON (b.key = d.key);
{\code}
and here is the stack trace ...
{code}
2013-07-21 10:14:04,440 INFO lazybinary.LazyBinaryStruct
(LazyBinaryStruct.java:parse(172)) - Missing fields! Expected 1 fields but only
got 0! Ignoring similar problems.
2013-07-21 10:14:04,444 FATAL ExecReducer (ExecReducer.java:reduce(269)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0) {"key":{"joinkey0":"128"},"value":{"_col1":""}}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:258)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:205)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:216)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:197)
at
org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:61)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
at
org.apache.hadoop.hive.ql.exec.MuxOperator.processOp(MuxOperator.java:249)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.DemuxOperator.processOp(DemuxOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:249)
... 3 more
{\code}
At here, DemuxOperator only replaces the tag (the tag seems correct at here)
and forward the row to MuxOperator. Then, MuxOperator will just forward the row
to the JoinOperator. Seems we cannot get the value in this case... Still
looking for the root cause... Seems related to MapJoin because the query with
hive.optimize.correlation=true and hive.auto.convert.join=false is fine.
> NPE - subquery smb joins fails
> ------------------------------
>
> Key: HIVE-4502
> URL: https://issues.apache.org/jira/browse/HIVE-4502
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.11.0
> Reporter: Vikram Dixit K
> Assignee: Navis
> Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
> HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
> smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that
> causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
