[
https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-20220:
--------------------------------------
Status: In Progress (was: Patch Available)
> Incorrect result when hive.groupby.skewindata is enabled
> --------------------------------------------------------
>
> Key: HIVE-20220
> URL: https://issues.apache.org/jira/browse/HIVE-20220
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 3.0.0
> Reporter: Ganesha Shreedhara
> Assignee: Ganesha Shreedhara
> Priority: Major
>
> hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped
> by keys to the reducers and hence avoids overloading a single reducer when
> there is a skew in data.
> This random distribution of keys is buggy when the reducer fails to fetch the
> mapper output due to a faulty datanode or any other reason. When reducer
> finds that it can't fetch mapper output, it sends a signal to Application
> Master to reattempt the corresponding map task. The reattempted map task will
> now get the different random value from rand function and hence the keys that
> gets distributed now to the reducer will not be same as the previous run.
>
> *Steps to reproduce:*
> create table test(id int);
> insert into test values
> (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9);
> SET hive.groupby.skewindata=true;
> SET mapreduce.reduce.reduces=2;
> //Add a debug port for reducer
> select count(1) from test group by id;
> //Remove mapper's intermediate output file when map stage is completed and
> one out of 2 reduce tasks is completed and then continue the run. This causes
> 2nd reducer to send event to Application Master to rerun the map task.
> The following is the expected result.
> 1
> 2
> 3
> 4
> 5
> 6
> 8
> 8
> 9
>
> But you may get different result due to a different value returned by the
> rand function in the second run causing different distribution of keys.
> This needs to be fixed such that the mapper distributes the same keys even if
> it is reattempted multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
