Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23363 )
Change subject: IMPALA-14349: Encode FileDescriptors in time in loading Iceberg Tables ...................................................................... Patch Set 1: (4 comments) Thanks for fixing this so quickly! I just have some minor comments. http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@111 PS1, Line 111: List<Pair<FileSystem, ContentFile<?>>> filesSupportsStorageIds = Lists.newArrayList(); It seems we are not using the FileSystems. Maybe we can simplify the type to List<ContentFile<?>>. http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@224 PS1, Line 224: collectPartitionPaths(contentFiles); Will this step take some time on large tables? Can we add a log showing how many partitions we collect and will list them? The log can also indicate whether this step is slow. http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@268 PS1, Line 268: MAX_HDFS_PARTITIONS_PARALLEL_LOAD Will non-HDFS storages use this method as well? If no, could you please add a comment? Or simply remove this method since it's used only once. http://gerrit.cloudera.org:8080/#/c/23363/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@296 PS1, Line 296: synchronized (numUnknownDiskIds) { Can we declare numUnknownDiskIds as AtomicLong to get rid of synchronized? -- To view, visit http://gerrit.cloudera.org:8080/23363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1c2a7119d76db7ce7c43caec2ccb122a014851b Gerrit-Change-Number: 23363 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Comment-Date: Wed, 03 Sep 2025 13:21:51 +0000 Gerrit-HasComments: Yes
