[
https://issues.apache.org/jira/browse/HIVE-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058517#comment-16058517
]
Thejas M Nair commented on HIVE-16918:
--------------------------------------
+1 to the patch. Let get this in asap, as without this the repl v2 is in a
broken state in master.
I think we should follow up on the pfile discussion. One concern on having
checks with "pfile" scheme, that i have is that "pfile" is a test only thing in
hive. What if someone introduce a "power file system" or so, that hive needs to
work with and they use "pfile" scheme ?
> Skip ReplCopyTask distcp for _metadata copying. Also enable -pb for distcp
> --------------------------------------------------------------------------
>
> Key: HIVE-16918
> URL: https://issues.apache.org/jira/browse/HIVE-16918
> Project: Hive
> Issue Type: Bug
> Components: repl
> Affects Versions: 3.0.0
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
> Attachments: HIVE-16918.2.patch, HIVE-16918.patch
>
>
> With HIVE-16686, we switched ReplCopyTask to always use a privileged DistCp.
> This, however, is incorrect for copying _metadata generated from a temporary
> scratch directory to hdfs. We need to change that so that routes to using a
> regular CopyTask. The issue with using distcp for this is that distcp
> launches from another job which may be queued on another machine, which does
> not have access to this file:// uri. Distcp should only ever be used when
> copying from non-localfilesystems.
> Also, in the spirit of following up HIVE-16686, we missed adding "-pb" as a
> default for invocations of distcp from hive. Adding that in. This would not
> be necessary if HADOOP-8143 had made it in, but till it doesn't go in, we
> need it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
