Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22559 )
Change subject: IMPALA-11402: Add limit on files fetched by a single getPartialCatalogObject request ...................................................................... Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/22559/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22559/1//COMMIT_MSG@16 PS1, Line 16: partition level when it's too big, and only return a subset of the > Reword this to so not say partitions are missing since that implies some ba Done. Hope it's clear now. http://gerrit.cloudera.org:8080/#/c/22559/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/22559/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1105 PS1, Line 1105: numFetchedParts, ids.size(), table); > Thanks for the clarification. Since the partial fetch is delimited by parti I previously thought this as a config error and let admin fix the config instead. However, the behavior is coordinator makes no progress here and goes into an infinite loop. It might be hard for the admin to notice this in time. So we should either fail here or let catalogd try to return at least one partition. I think failing here is better. Otherwise, the config is not a hard limit and can't protect catalogd from hitting OOM error and got killed unintentionally by JVM options like "-XX:+ExitOnOutOfMemoryError". Added a check here that will fail if no partitions are returned. Added test coverage for this. -- To view, visit http://gerrit.cloudera.org:8080/22559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb13fec20de5a17e7fc33613ca5cdebb9ac1a1e5 Gerrit-Change-Number: 22559 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Thu, 20 Mar 2025 02:20:19 +0000 Gerrit-HasComments: Yes
