Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/20700 )
Change subject: IMPALA-12299: Parallelize file listings of Iceberg tables on HDFS/Ozone ...................................................................... Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/20700/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20700/2//COMMIT_MSG@17 PS2, Line 17: : - 280.0 > Could you please elaborate on this a little more? I think it's worth to hig Done http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@206 PS2, Line 206: n > nit: for single-line if statements we omit the braces. Done http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@239 PS2, Line 239: partitionPaths = icebergFilesByPartition(); > Can we just invoke icebergFilesByPartition() in the current thread? Submitt It's left it in by mistake, I did some experimenting with spliterator parallelizing, but it's it seemed that the overhead for splitting the tasks was not worth it. http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@247 PS2, Line 247: > nit: for readability, it worth to write it in its own line. Done http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/20700/2/fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java@30 PS2, Line 30: import javax.annotation.Nullable; > The new imports seem to be unused. Done -- To view, visit http://gerrit.cloudera.org:8080/20700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5ca7e873f4ad0cc8dab6a77b62e05d965b4a76d Gerrit-Change-Number: 20700 Gerrit-PatchSet: 3 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 23 Nov 2023 13:54:36 +0000 Gerrit-HasComments: Yes
