Steve Loughran created HADOOP-16546:
---------------------------------------
Summary: make sure staging committers collect DTs for the staging
FS
Key: HADOOP-16546
URL: https://issues.apache.org/jira/browse/HADOOP-16546
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.2.0
Reporter: Steve Loughran
This is not a problem I've seen in the wild, but I've now encountered a problem
with hive doing something like this
we need to (somehow) make sure that the staging committers collect DTs for the
staging dir FS. If this is the default FS or the same as a source or dest FS,
this is handled elsewhere, but otherwise we need to add the staging fs.
I don;t see an easy way to do this, but we could add a new method to
PathOutputCommitter to collect DTs; FileOutputFormat can invoke this alongside
its ongoing collection of tokens for the output FS. Base impl would be a no-op,
obviously.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
