Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/22559 )
Change subject: IMPALA-11402: Add limit on files fetched by a single getPartialCatalogObject request ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/22559/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22559/1//COMMIT_MSG@28 PS1, Line 28: Choose 4000000 as the default value for this new flag to leave some > That's a good point. The majority of the size of a file descriptor is the f That's a pretty significant difference. If that is the overall time, we should probably lower the size on this evidence alone . Having smaller RPCs may also help reduce contention with other clients. http://gerrit.cloudera.org:8080/#/c/22559/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/22559/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1105 PS1, Line 1105: List<Long> remainingIds = Lists.newArrayListWithCapacity( Can this be bounded by backendCfg_.catalog_partial_fetch_max_files to not pass all of the remaining ids on every call? -- To view, visit http://gerrit.cloudera.org:8080/22559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb13fec20de5a17e7fc33613ca5cdebb9ac1a1e5 Gerrit-Change-Number: 22559 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Fri, 14 Mar 2025 17:36:32 +0000 Gerrit-HasComments: Yes
