[
https://issues.apache.org/jira/browse/HIVE-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113308#comment-15113308
]
Ashutosh Chauhan commented on HIVE-12897:
-----------------------------------------
Currently, Hive::loadDynamicPartitions() makes N number of calls to metastore
just to check whether partition name is valid against configured white list
pattern. It also requests all file names under all partition dirs which is also
unnecessary. This patch:
* Only request dir names from NN instead of all files under it.
* It moved partition name check to FS operator, so that instead of waiting
doing all the work and then throwing exception at end of it, we do it as soon
as possible. Thus avoiding wasted work on cluster.
* Because of above It completely gets rid of N calls to metastore, where N =
all files created under all partition during the job.
Patch is ready for review.
> Improve dynamic partition loading
> ---------------------------------
>
> Key: HIVE-12897
> URL: https://issues.apache.org/jira/browse/HIVE-12897
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Attachments: HIVE-12897.2.patch, HIVE-12897.patch
>
>
> There are many redundant calls to metastore which is not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
