[
https://issues.apache.org/jira/browse/HIVE-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-12563:
------------------------------
Attachment: HIVE-12563.1.patch
Not sure if this is the right fix, but looks like perhaps we should be using
fetchNextGroup() rather than promoteNextGroupToCandidate() during
joinFinalLeftData(), in case the logic tries to fetch another row.
Can you take a look [~vikram.dixit] [~hagleitn]
> NullPointerException with 3-way Tez merge join
> ----------------------------------------------
>
> Key: HIVE-12563
> URL: https://issues.apache.org/jira/browse/HIVE-12563
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 1.2.0
> Reporter: Jason Dere
> Attachments: HIVE-12563.1.patch
>
>
> Issue occurs in Tez merge joins with 3 way join (A join B join C), where A is
> the big table, and A and C end up having 0 rows after table filters.
> Was able to repro this issue with using the sample tables in the Hive unit
> tests:
> {noformat}
> select
> a.key, b.value, c.value
> from
> src a,
> src1 b,
> src c
> where
> a.key = b.key and a.key = c.key
> and a.value > 'wal_6789'
> and c.value > 'wal_6789'
> {noformat}
> {noformat}
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime
> Error while closing operators:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:162)
> ... 13 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:312)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:293)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:501)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:416)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:287)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:439)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:407)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:310)
> ... 19 more
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:431)
> ... 21 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=1)
> {"key":{"reducesinkkey0":"146"},"value":{"_col0":"val_146"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:292)
> ... 22 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:288)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:361)
> ... 23 more
> ]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex
> vertex_1449041190339_0001_1_03 [Reducer 2] killed/failed due to:null]DAG
> failed due to vertex failure. failedVertices:1 killedVertices:0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
