[
https://issues.apache.org/jira/browse/IMPALA-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118256#comment-18118256
]
Arnab Karmakar commented on IMPALA-13449:
-----------------------------------------
Thanks for sharing the jira!
Here are some key findings on large tables. The current formula always
*overestimates* actual retained heap. The original jira mentioned
underestimation, that may still apply to Iceberg or other table types we have
not calibrated here. For external HDFS/cloud tables, the data points the other
way.
Current est vs MAT on large tables:
||Shape||HDFS MAT||Prod gap||S3 MAT||Prod gap||
|P==F (500K/500K)|821 MB|+57%|591 MB|+106%|
|P<<F (1/500K)|74 MB|+317%|44 MB|+443%|
|P>>F (500K/1)|538 MB|+81%|539 MB|+81%|
The current formula treats partition and file slots at similar marginal cost
(~2048 vs ~500 B). MAT shows file/block metadata is much cheaper at scale when
many files share few partitions (~78–92 B/file on HDFS/cloud multifile), while
partition metadata in sparse tables is ~1,130 B/partition, much lower than the
2048 B constant but still the dominant term when P>>F.
Since the previous idea of using a scale factor didn't work quite well, Ive
tried calibrating the constants and the formula for cloud. The error rate with
the proposed constants is higher at smaller scale but converges to the actual
numbers as scale increases.
Proposed constants:
||FS||Formula||Worst-case gap (at 500K scale)||
|Cloud|{{1146·P + 92·F}}|±1.4%|
|HDFS (all shapes)|{{1293·P + 89·(F+B)}}|±15%|
|HDFS (considering P>>F is rare)|{{1565·P + 78·(F+B)}}|+39% on P >> F; ±0% on
1:1 + multifile|
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-13449
> URL: https://issues.apache.org/jira/browse/IMPALA-13449
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Manish Maheshwari
> Assignee: Arnab Karmakar
> Priority: Critical
>
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables. The current size shows tables extremely small and the logic needs to
> be refreshed for Iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
