[
https://issues.apache.org/jira/browse/IMPALA-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118715#comment-18118715
]
Quanlong Huang commented on IMPALA-13449:
-----------------------------------------
Agree that the current constants are too large.
For file descriptors, I think the file name (not absolute file path) length
also matters. In the example shown in
[IMPALA-14583|https://issues.apache.org/jira/browse/IMPALA-14583?focusedCommentId=18044376&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18044376],
the file name (500_cols.parq) is short. I guess that's why you see each file
needs less than 100B. However, in IMPALA-13177, I saw much longer file names
generated by Spark jobs, which results in larger FileDescriptors (encoded into
160B).
The FlatBuffer definition of FbFileDesc has two string fields, relative_path
and absolute_path(only used for external files of Iceberg tables):
https://github.com/apache/impala/blob/d757d3dfe5c7cdcc48aed2064ed28bbed3ef9ffe/common/fbs/CatalogObjects.fbs#L64
So instead of using a fixed estimation for FileDescriptor, we should consider
the average file name length. We can get this during metadata loading where we
list the files, and add such field in FileMetadataStats.
https://github.com/apache/impala/blob/d757d3dfe5c7cdcc48aed2064ed28bbed3ef9ffe/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java#L129
For partitions and HDFS blocks, could you investigate why their sizes change in
different shapes? Updating PER_BLOCK_MEM_USAGE_BYTES from 150 to 100 looks good
to me since FbFileBlock just have numeric values. But for partitions, we need
to check if there are var-length stuffs like file names in FbFileDesc.
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-13449
> URL: https://issues.apache.org/jira/browse/IMPALA-13449
> Project: IMPALA
> Issue Type: New Feature
> Components: Catalog
> Reporter: Manish Maheshwari
> Assignee: Arnab Karmakar
> Priority: Critical
>
> Improve / Correct the Table Size in Catalog Web UI for External and Iceberg
> Tables. The current size shows tables extremely small and the logic needs to
> be refreshed for Iceberg tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
