Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23363 )
Change subject: IMPALA-14349: Encode FileDescriptors in time in loading Iceberg Tables ...................................................................... Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@224 PS1, Line 224: private List<IcebergFileDescriptor> parallelListing( > collectPartitionPaths() takes some time indeed: Thanks for the experiment! http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@296 PS1, Line 296: Reference<Long> localNumUnknownDiskIds = new Reference<>(0L); > I can do it, I only hesitate because it will require changing quite a few > method signatures elsewhere, just because of IcebergFileMetadataLoader. I think we just need to change two methods in IcebergFileMetadataLoader: parallelListing() and createFdsForPartition(). For 'localNumUnknownDiskIds' , we can keep it as-is since it's accessed by a single thread. Using AtomicLong or AtomicInteger also simplify the code a little bit. -- To view, visit http://gerrit.cloudera.org:8080/23363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1c2a7119d76db7ce7c43caec2ccb122a014851b Gerrit-Change-Number: 23363 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Sun, 07 Sep 2025 08:50:11 +0000 Gerrit-HasComments: Yes
