[
https://issues.apache.org/jira/browse/HIVE-27899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788725#comment-17788725
]
Chenyu Zheng commented on HIVE-27899:
-------------------------------------
The probability of this bug recurring is very very low, and some special code
must be added to increase the probability of this bug recurring. Only then can
the correctness of the repair code be guaranteed. I added a sleep to the
relevant code to simulate the stuck. This greatly increases the probability of
the problem recurring. I've added the relevant details in the attachment
`reproduce_bug.md`.
> Killed speculative execution task attempt should not commit file
> ----------------------------------------------------------------
>
> Key: HIVE-27899
> URL: https://issues.apache.org/jira/browse/HIVE-27899
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Chenyu Zheng
> Assignee: Chenyu Zheng
> Priority: Major
> Attachments: reproduce_bug.md
>
>
> As I mentioned in HIVE-25561, when tez turns on speculative execution, the
> data file produced by hive may be duplicated. I mentioned in HIVE-25561 that
> if the speculatively executed task is killed, some data may be submitted
> unexpectedly. However, after HIVE-25561, there is still a situation that has
> not been solved. If two task attempts commit file at the same time, the
> problem of duplicate data files may also occur. Although the probability of
> this happening is very, very low, it does happen.
>
> Why?
> There are two key steps:
> (1)FileSinkOperator::closeOp
> TezProcessor::initializeAndRunProcessor --> ... --> FileSinkOperator::closeOp
> --> fsp.commit
> When the OP is closed, the process of closing the OP will be triggered, and
> eventually the call to fsp.commit will be triggered.
> (2)removeTempOrDuplicateFiles
> (2.a)Firstly, listStatus the files in the temporary directory.
> (2.b)Secondly check whether there are multiple incorrect commit, and finally
> move the correct results to the final directory.
> When speculative execution is enabled, when one attempt of a Task is
> completed, other attempts will be killed. However, AM only sends the kill
> event and does not ensure that all cleanup actions are completed, that is,
> closeOp may be executed between 2.a and 2.b. Therefore,
> removeTempOrDuplicateFiles will not delete the file generated by the kill
> attempt.
> How?
> The problem is that both speculatively executed tasks commit the file. This
> will not happen in the Tez examples because they will try canCommit, which
> can guarantee that one and only one task attempt commit successfully. If one
> task attempt executes canCommti successfully, the other one will be stuck by
> canCommti until it receives a kill signal.
> detail see:
> https://github.com/apache/tez/blob/51d6f53967110e2b91b6d90b46f8e16bdc062091/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/SimpleMRProcessor.java#L70
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
