[
https://issues.apache.org/jira/browse/HIVE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593138#comment-15593138
]
Steve Loughran commented on HIVE-14271:
---------------------------------------
one funny about last-writer-wins is the scenario
# executor 1 starts working on part-001
# executor 2 gets starts working on it, also opens stream to part-001
# executor 2 finishes; their work becomes visible
# whatever was waiting for part 001 to be ready sets off
# executor 1 finishes and overwrites the existing part 001
That needs to be avoided
> FileSinkOperator should not rename files to final paths when S3 is the
> default destination
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-14271
> URL: https://issues.apache.org/jira/browse/HIVE-14271
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sergio Peña
> Assignee: Sergio Peña
>
> FileSinkOperator does a rename of {{outPaths -> finalPaths}} when it finished
> writing all rows to a temporary path. The problem is that S3 does not support
> renaming.
> Two options can be considered:
> a. Use a copy operation instead. After FileSinkOperator writes all rows to
> outPaths, then the commit method will do a copy() call instead of move().
> b. Write row by row directly to the S3 path (see HIVE-1620). This may add
> better performance calls, but we should take care of the cleanup part in case
> of writing errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
