[
https://issues.apache.org/jira/browse/HIVE-29775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29775:
--------------------------------
Description:
{{Utilities.[moveFile|https://github.com/apache/hive/blob/5e9f78815688457ff00d7b3192f7e13d458a16d6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1145-L1158]}}
uses the same "probe {{exists()}} -> pick {{_copy_N}} -> {{{}rename(){}}}"
pattern that HIVE-28822 fixes in {{{}Hive.mvFile{}}}. On filesystems whose
rename is not atomic-if-absent (S3A and other object stores), two concurrent
writers can pick the same destination between the probe and the rename;
outcomes are the same two failure modes:
* *Fail-loud* — {{S3AFileSystem.initiateRename}} throws
{{{}FileAlreadyExistsException{}}}, which {{Utilities.moveFile}} wraps into a
{{HiveException("Unable to move: ... to: ...")}} and hands back to the caller.
* *Fail-silent* — both writers' internal {{exists()}} probes see the target as
not-yet-present, both PUTs go to the same key, and the second silently
overwrites the first (last writer wins, no error surfaces).
HIVE-28822 fixed only {{{}Hive.mvFile{}}}. This ticket applies the same
approach to {{Utilities.moveFile}} (and the related helpers below).
was:
{{Utilities.[moveFile|https://github.com/apache/hive/blob/5e9f78815688457ff00d7b3192f7e13d458a16d6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1145-L1158]}}
uses the same "probe {{exists()}} -> pick {{_copy_N}} -> {{{}rename(){}}}"
pattern that HIVE-28822 just fixed in {{{}Hive.mvFile{}}}. On filesystems whose
rename is not atomic-if-absent (S3A and other object stores), two concurrent
writers can pick the same destination between the probe and the rename;
outcomes are the same two failure modes HIVE-28822 addressed:
* *Fail-loud* — {{S3AFileSystem.initiateRename}} throws
{{{}FileAlreadyExistsException{}}}, which {{Utilities.moveFile}} wraps into a
{{HiveException("Unable to move: ... to: ...")}} and hands back to the caller.
* *Fail-silent* — both writers' internal {{exists()}} probes see the target as
not-yet-present, both PUTs go to the same key, and the second silently
overwrites the first (last writer wins, no error surfaces).
HIVE-28822 fixed only {{{}Hive.mvFile{}}}. This ticket applies the same
approach to {{Utilities.moveFile}} (and the related helpers below).
> Fix a possible exists() - pick copy_N - rename race condition in
> Utilities.moveFile to avoid possible data loss
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29775
> URL: https://issues.apache.org/jira/browse/HIVE-29775
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Priority: Major
>
> {{Utilities.[moveFile|https://github.com/apache/hive/blob/5e9f78815688457ff00d7b3192f7e13d458a16d6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1145-L1158]}}
> uses the same "probe {{exists()}} -> pick {{_copy_N}} -> {{{}rename(){}}}"
> pattern that HIVE-28822 fixes in {{{}Hive.mvFile{}}}. On filesystems whose
> rename is not atomic-if-absent (S3A and other object stores), two concurrent
> writers can pick the same destination between the probe and the rename;
> outcomes are the same two failure modes:
> * *Fail-loud* — {{S3AFileSystem.initiateRename}} throws
> {{{}FileAlreadyExistsException{}}}, which {{Utilities.moveFile}} wraps into a
> {{HiveException("Unable to move: ... to: ...")}} and hands back to the caller.
> * *Fail-silent* — both writers' internal {{exists()}} probes see the target
> as not-yet-present, both PUTs go to the same key, and the second silently
> overwrites the first (last writer wins, no error surfaces).
> HIVE-28822 fixed only {{{}Hive.mvFile{}}}. This ticket applies the same
> approach to {{Utilities.moveFile}} (and the related helpers below).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
