Quanlong Huang created IMPALA-14495:
---------------------------------------
Summary: Dangling inflight event ids added during metadata loading
Key: IMPALA-14495
URL: https://issues.apache.org/jira/browse/IMPALA-14495
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
While enabling TRACE logging of catalogd, I found some strange logs during
metadata loading:
{noformat}
I20251014 09:24:55.444000 334364 MetaStoreUtil.java:191] Fetching 24 partitions
for: functional.alltypes using partition batch size: 1000
I20251014 09:24:55.487201 334364 MetaStoreUtil.java:223] Fetched 24/24
partitions for table functional.alltypes
I20251014 09:24:55.487264 334364 HdfsTable.java:1314] Fetched partition
metadata from the Metastore: functional.alltypes
I20251014 09:24:55.508351 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.516407 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.516611 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.516772 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.516906 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.517033 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.517458 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.517647 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.517796 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.517954 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518092 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518218 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518339 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518450 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518558 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518667 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518779 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.518888 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519011 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519126 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519240 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519448 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519563 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.519682 334364 HdfsPartition.java:1283] Added 12277 to
inflight events 12277
I20251014 09:24:55.521536 334364 ParallelFileMetadataLoader.java:230] Loading
file and block metadata for 24 paths for table functional.alltypes using a
thread pool of size 5
{noformat}
This is a cold start of catalogd and there are no DDL/DMLs executed yet. The
event id 12277 is from a previous run and shouldn't be tracked as inflight
event id.
I think the problem is that we don't check the catalog service id and use the
catalog version in hms parameters directly:
https://github.com/apache/impala/blob/e7aa31296c8cb51572992294ab4fec8eb3d8541c/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java#L1281-L1282
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
