[
https://issues.apache.org/jira/browse/IMPALA-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115533#comment-18115533
]
Arnab Karmakar commented on IMPALA-13449:
-----------------------------------------
Thanks for the tip [~stigahuang], wanted to sanity-check whether this approach
for estimating mem usage is worth pursuing further.
{*}TL;DR{*}: I think I've found a clean fix for the catalog metadata memory
estimate on non-Iceberg hdfs tables.
Problem: the current P×2048 + F×500 + B×150 formula isn't just mis-tuned, it's
somewhat unable to be right across table sizes.
Heap dumps show the real retained bytes/partition + file slot shrink as tables
grow: *~2,229 B/slot* at 24 partitions (functional.alltypes) vs *~1,331 B/slot*
at 1,824 (tpcds.store_sales). So no re-calibration of the constants can fit
both ends. Tried tuning for small tables and big ones overestimate and while
tuning for big tables, the small ones underestimate.
{*}Proposed fix{*}: keep every existing constant and multiply the partition +
file term by one decay factor {*}S(P) = A × numPartitions^(−E){*}. Across 12
heap-dump reference tables this takes mean error from {*}~32% down to ~3%,
worst case 91% -> <6%{*}.
|Config (table size / FS / file format) |MAT retained (B)|Current
gap|Scaled gap|
|small table / cloud / parquet| 53,504| +14.3%| +3.6%|
|small table / cloud / text | 55,600| +10.0%| −0.3%|
|small table / hdfs / parquet | 63,176| +2.5%| +5.4%|
|small table / hdfs / text | 69,696| −7.1%| −4.4%|
|large table / cloud / parquet| 2,428,424| +91.4%| −1.2%|
|large table / cloud / text | 2,522,912| +84.2%| −4.9%|
|large table / hdfs / parquet | 3,595,968| +36.9%| −0.2%|
|large table / hdfs / text | 3,795,264| +29.7%| −5.5%|
Two things I want to flag up front:
- Its not going to conflict with the Iceberg work as IcebergTable extends Table
(not HdfsTable) with its own updateMetrics(), and this change doesn't alter the
shared PER_FD/PER_BLOCK values. (However, calibrating constants might conflict)
- I've only validated up to 1,824 partitions; I've extrapolated to ~1M and it
stays reasonable (579 B/slot on cloud and 1,063B/slot on hdfs).
After all the considerations, just fixing the constants alone wont correct the
estimates for tables of all scales and FileSystems.
If you think the scale factor approach is worth exploring further, happy to
write and share a formal doc for better clarity.
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-13449
> URL: https://issues.apache.org/jira/browse/IMPALA-13449
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Manish Maheshwari
> Assignee: Arnab Karmakar
> Priority: Critical
>
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables. The current size shows tables extremely small and the logic needs to
> be refreshed for Iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
