[
https://issues.apache.org/jira/browse/HIVE-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409287#comment-15409287
]
Rajesh Balamohan commented on HIVE-14423:
-----------------------------------------
[~cnauroth] - Thanks for the details. Yes, I used listStatus with recursive
option as it would be tuned with the patches Steve is working on. But need to
check if this is going to hurt HDFS performance due to number of network calls.
I have not done that experiment yet. It would be good to have an optimized
version of getContentSummary in S3A. I will share the revised patch.
> S3: Fetching partition sizes from FS can be expensive when stats are not
> available in metastore
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-14423
> URL: https://issues.apache.org/jira/browse/HIVE-14423
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: HIVE-14423.1.patch
>
>
> When partition stats are not available in metastore, it tries to get the file
> sizes from FS.
> e.g
> {noformat}
> at
> org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
> at
> org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> {noformat}
> This can be quite expensive in some FS like S3. Especially when table is
> partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can
> spend 1000s of seconds just waiting for these information to be pulled in.
> Also, it would be good to remove FS.getContentSummary usage to find out file
> sizes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
