Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21762 )
Change subject: IMPALA-13333: Limit memory estimation if PlanNode can spill ...................................................................... Patch Set 37: (2 comments) http://gerrit.cloudera.org:8080/#/c/21762/37/fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java File fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java: http://gerrit.cloudera.org:8080/#/c/21762/37/fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java@103 PS37, Line 103: memEstimateBytes_ = Math.min(memEstimateBytes_, scaleBasedMemEstimate); > For the case where `minMemReservationBytes_ > maxMemEstimateBytesAfterScali Agree. Lets not change memEstimateBytes_ in that case. http://gerrit.cloudera.org:8080/#/c/21762/37/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/21762/37/fe/src/main/java/org/apache/impala/service/Frontend.java@2136 PS37, Line 2136: new_entry.setMax_mem_limit( > Ideally, we want a logic similar to Analyzer.getMaxMemLimitPerHost(false) t I think this should revert to what it used to be. The latest code in Analyzer.java look up value from TPoolConfig rather than TExecutorGroupSet from here. That leave TPoolConfig.getMax_mem_limit() calls in Frontend.java that only relevant for workload-aware autoscaling (sorting executor group, etc) that should not mix MEM_LIMIT options in its calculation. -- To view, visit http://gerrit.cloudera.org:8080/21762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I290c4e889d4ab9e921e356f0f55a9c8b11d0854e Gerrit-Change-Number: 21762 Gerrit-PatchSet: 37 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Reviewer: Yida Wu <wydbaggio...@gmail.com> Gerrit-Comment-Date: Wed, 12 Mar 2025 02:02:34 +0000 Gerrit-HasComments: Yes
