[
https://issues.apache.org/jira/browse/HIVE-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576937#comment-16576937
]
Hive QA commented on HIVE-20220:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935173/HIVE-20220.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14875 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13155/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13155/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13155/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935173 - PreCommit-HIVE-Build
> Incorrect result when hive.groupby.skewindata is enabled
> --------------------------------------------------------
>
> Key: HIVE-20220
> URL: https://issues.apache.org/jira/browse/HIVE-20220
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 3.0.0
> Reporter: Ganesha Shreedhara
> Assignee: Ganesha Shreedhara
> Priority: Major
> Attachments: HIVE-20220.2.patch, HIVE-20220.patch
>
>
> hive.groupby.skewindata makes use of rand UDF to randomly distribute grouped
> by keys to the reducers and hence avoids overloading a single reducer when
> there is a skew in data.
> This random distribution of keys is buggy when the reducer fails to fetch the
> mapper output due to a faulty datanode or any other reason. When reducer
> finds that it can't fetch mapper output, it sends a signal to Application
> Master to reattempt the corresponding map task. The reattempted map task will
> now get the different random value from rand function and hence the keys that
> gets distributed now to the reducer will not be same as the previous run.
>
> *Steps to reproduce:*
> create table test(id int);
> insert into test values
> (1),(2),(2),(3),(3),(3),(4),(4),(4),(4),(5),(5),(5),(5),(5),(6),(6),(6),(6),(6),(6),(7),(7),(7),(7),(7),(7),(7),(7),(8),(8),(8),(8),(8),(8),(8),(8),(9),(9),(9),(9),(9),(9),(9),(9),(9);
> SET hive.groupby.skewindata=true;
> SET mapreduce.reduce.reduces=2;
> //Add a debug port for reducer
> select count(1) from test group by id;
> //Remove mapper's intermediate output file when map stage is completed and
> one out of 2 reduce tasks is completed and then continue the run. This causes
> 2nd reducer to send event to Application Master to rerun the map task.
> The following is the expected result.
> 1
> 2
> 3
> 4
> 5
> 6
> 8
> 8
> 9
>
> But you may get different result due to a different value returned by the
> rand function in the second run causing different distribution of keys.
> This needs to be fixed such that the mapper distributes the same keys even if
> it is reattempted multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
