[
https://issues.apache.org/jira/browse/IMPALA-13800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13800:
------------------------------------
Issue Type: Improvement (was: Bug)
> Add catalogd logs for the progress of file metadata loading
> -----------------------------------------------------------
>
> Key: IMPALA-13800
> URL: https://issues.apache.org/jira/browse/IMPALA-13800
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> When loading file metadata of an Hdfs table, we just have logs when the
> parallel load starts and finishes:
> {code:java}
> I0224 23:59:51.042261 2189 HdfsTable.java:1366] Fetched partition metadata
> from the Metastore: default.part_6m
> I0225 00:01:46.838979 2189 ParallelFileMetadataLoader.java:217] Loading file
> and block metadata for 6000000 paths for table default.part_6m using a thread
> pool of size 16
> I0225 00:05:38.324651 2189 HdfsTable.java:840] Loaded file and block
> metadata for default.part_6m partitions: p=0/q=0, p=0/q=1, p=0/q=10, and
> 5999997 others. Time taken: 4m6s
> W0225 00:05:45.111688 2189 HdfsTable.java:1388] Time taken on loading table
> default.part_6m exceeded warning threshold. Time: 11m43s{code}
> After fetching the partition metadata from HMS, no other logs except the JVM
> pause logs are shown. It'd be helpful to add more logs to show the progress.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
