[
https://issues.apache.org/jira/browse/HIVE-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529673#comment-14529673
]
Peter Slawski commented on HIVE-10538:
--------------------------------------
The Spark driver failures are caused by this change. This would be expected if
a row's hashcode affected its ordering in Spark. This patch makes it so that
HiveKey's hashcode outputted from ReduceSinkOperator is no longer always
multiplied by 31 (as explained previously).
Also, for at least those failed qtests, the row ordering/output in the expected
output differs across MapRed, Tez, and Spark. So, execution engine affects
ordering.
>From
>[spark/groupby_complex_types_multi_single_reducer.q.out#L221|https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/spark/groupby_complex_types_multi_single_reducer.q.out#L221]
{code}
POSTHOOK: query: SELECT DEST2.* FROM DEST2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@dest2
#### A masked pattern was here ####
{"120":"val_120"} 2
{"129":"val_129"} 2
{"160":"val_160"} 1
{"26":"val_26"} 2
{"27":"val_27"} 1
{"288":"val_288"} 2
{"298":"val_298"} 3
{"30":"val_30"} 1
{"311":"val_311"} 3
{"74":"val_74"} 1
{code}
>From
>[groupby_complex_types_multi_single_reducer.q.out#L240|https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/groupby_complex_types_multi_single_reducer.q.out#L240]
{code}
POSTHOOK: query: SELECT DEST2.* FROM DEST2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@dest2
#### A masked pattern was here ####
{"0":"val_0"} 3
{"10":"val_10"} 1
{"100":"val_100"} 2
{"103":"val_103"} 2
{"104":"val_104"} 2
{"105":"val_105"} 1
{"11":"val_11"} 1
{"111":"val_111"} 1
{"113":"val_113"} 2
{"114":"val_114"} 1
{code}
> Fix NPE in FileSinkOperator from hashcode mismatch
> --------------------------------------------------
>
> Key: HIVE-10538
> URL: https://issues.apache.org/jira/browse/HIVE-10538
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 1.0.0, 1.2.0
> Reporter: Peter Slawski
> Assignee: Peter Slawski
> Priority: Critical
> Fix For: 1.2.0, 1.3.0
>
> Attachments: HIVE-10538.1.patch, HIVE-10538.1.patch,
> HIVE-10538.1.patch
>
>
> A Null Pointer Exception occurs when in FileSinkOperator when using bucketed
> tables and distribute by with multiFileSpray enabled. The following snippet
> query reproduces this issue:
> {code}
> set hive.enforce.bucketing = true;
> set hive.exec.reducers.max = 20;
> create table bucket_a(key int, value_a string) clustered by (key) into 256
> buckets;
> create table bucket_b(key int, value_b string) clustered by (key) into 256
> buckets;
> create table bucket_ab(key int, value_a string, value_b string) clustered by
> (key) into 256 buckets;
> -- Insert data into bucket_a and bucket_b
> insert overwrite table bucket_ab
> select a.key, a.value_a, b.value_b from bucket_a a join bucket_b b on (a.key
> = b.key) distribute by key;
> {code}
> The following stack trace is logged.
> {code}
> 2015-04-29 12:54:12,841 FATAL [pool-110-thread-1]: ExecReducer
> (ExecReducer.java:reduce(255)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{},"value":{"_col0":"113","_col1":"val_113"}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.findWriterOffset(FileSinkOperator.java:819)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:747)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:235)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
