Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22559 )
Change subject: IMPALA-11402: Add limit on files fetched by a single getPartialCatalogObject request ...................................................................... Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/22559/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/22559/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2356 PS9, Line 2356: "for table {}. Coordinator should fetch the remaining partitions " + > Error looks good now. I don't think the warning is good though since this i This warning indicates the table currently has too many partitions or too many files. Users should optimize it by compacting small files or clearing old partitions. They can grep for this log to find such tables. The above ERROR log is rare to show up since a single partition (or non-partitioned table) usually don't have that many files. This warning also indicates coordinator sends multiple requests to fetch the partition metadata, which results in longer planning time. Usually coordinator just sends one request to fetch all missing partitions of a table. So it's a warning about performance. -- To view, visit http://gerrit.cloudera.org:8080/22559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb13fec20de5a17e7fc33613ca5cdebb9ac1a1e5 Gerrit-Change-Number: 22559 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Sun, 06 Apr 2025 10:45:23 +0000 Gerrit-HasComments: Yes
