-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50788/
-----------------------------------------------------------
(Updated Aug. 4, 2016, 11:08 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-14423
https://issues.apache.org/jira/browse/HIVE-14423
Repository: hive-git
Description (updated)
-------
When partition stats are not available in metastore, it tries to get the file
sizes from FS.
e.g
at
org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
at
org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
This can be quite expensive in some FS like S3. Especially when table is
partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can
spend 1000s of seconds just waiting for these information to be pulled in.
Also, it would be good to remove FS.getContentSummary usage to find out file
sizes.
With patch, timing comes down to ~80 seconds for file size gathering instead of
1000s of seconds.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f5f619
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 2a9dc11
Diff: https://reviews.apache.org/r/50788/diff/
Testing
-------
Thanks,
Rajesh Balamohan
