[
https://issues.apache.org/jira/browse/HIVE-11456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658961#comment-14658961
]
Sushanth Sowmyan commented on HIVE-11456:
-----------------------------------------
Thanks for the fix - I have an additional question to verify if this causes a
problem. In the case of appends, where a previous file already exists, it's
possible that HCat would add an additional suffix to the resultant file, as
noted by the following:
https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L650-L656
I want to make sure that this is not a surprise to you, and is okay?
> HCatStorer should honor mapreduce.output.basename
> -------------------------------------------------
>
> Key: HIVE-11456
> URL: https://issues.apache.org/jira/browse/HIVE-11456
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.2.0
> Reporter: Rohini Palaniswamy
> Assignee: Mithun Radhakrishnan
> Priority: Critical
> Fix For: 1.3.0, 1.2.1, 2.0.0
>
> Attachments: HIVE-11456.1.patch
>
>
> Pig on Tez scripts with union directly followed by HCatStorer have a problem
> due to HCatStorer not honoring mapreduce.output.basename and always using
> part. Tez sets mapreduce.output.basename to part-v000-o000 (vertex id
> followed by output id). With union optimizer, Pig uses vertex groups to write
> directly from both the vertices to the final output directory. Since hcat
> ignores the mapreduce.output.basename, both the vertices produce
> part-r-0000<n> and when they are moved from the temp location to the final
> directory, they just overwrite each other. There is no failure and only one
> of the files with that name makes it into the final directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
