[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-5011:
-----------------------------------
Description:
Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500
trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal "hive-style" partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify "patterns"(like, say "$\{rootdir\}/$v1.$v2/") into which
writes happen, only "locations", and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way dynamic
ptn. code tries to determine what was written to where from the output
committer, which means that even if we supported patterned-writes instead of
location-writes, we still have to do some more deep diving into the output
committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths for
external tables *ONLY* for static partition writes - i.e., if we can determine
that the write is a dyn. ptn. write, we will ignore the user specification.
(Note that this does not mean we ignore the table's external location - we
honour that - we just don't honour any HCatStorer/etc provided additional
location - we stick to what metadata tells us the root location is.
was:
Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500
trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal "hive-style" partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify "patterns"(like, say "${rootdir}/$v1.$v2/") into which writes
happen, only "locations", and the values for dyn. partitions are not known
ahead of time. Also, specifying a custom path messes with the way dynamic ptn.
code tries to determine what was written to where from the output committer,
which means that even if we supported patterned-writes instead of
location-writes, we still have to do some more deep diving into the output
committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths for
external tables *ONLY* for static partition writes - i.e., if we can determine
that the write is a dyn. ptn. write, we will ignore the user specification.
(Note that this does not mean we ignore the table's external location - we
honour that - we just don't honour any HCatStorer/etc provided additional
location - we stick to what metadata tells us the root location is.
> Dynamic partitioning in HCatalog broken on external tables
> ----------------------------------------------------------
>
> Key: HIVE-5011
> URL: https://issues.apache.org/jira/browse/HIVE-5011
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
>
> Dynamic partitioning with HCatalog has been broken as a result of
> HCATALOG-500 trying to support user-set paths for external tables.
> The goal there was to be able to support other custom destinations apart from
> the normal "hive-style" partitions. However, it is not currently possible for
> users to set paths for dynamic ptn writes, since we don't support any way for
> users to specify "patterns"(like, say "$\{rootdir\}/$v1.$v2/") into which
> writes happen, only "locations", and the values for dyn. partitions are not
> known ahead of time. Also, specifying a custom path messes with the way
> dynamic ptn. code tries to determine what was written to where from the
> output committer, which means that even if we supported patterned-writes
> instead of location-writes, we still have to do some more deep diving into
> the output committer code to support it.
> Thus, my current proposal is that we honour writes to user-specified paths
> for external tables *ONLY* for static partition writes - i.e., if we can
> determine that the write is a dyn. ptn. write, we will ignore the user
> specification. (Note that this does not mean we ignore the table's external
> location - we honour that - we just don't honour any HCatStorer/etc provided
> additional location - we stick to what metadata tells us the root location is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
