Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG@16 PS1, Line 16: data_stream_sender_buffer_size_used_by_planner > Why not utilize data_stream_sender_buffer_size flag straight away? I thought that it can be useful to use a slightly smaller / larger value if this have to be tuned. + a new flag was needed anyway, as I didn't want this to be the default http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java: http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@100 PS1, Line 100: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize)); > If getAvgSerializedRowSize() is significantly larger than getFixedLenRowSiz Even if use_be_logic is true, the calculation still uses getAvgSerializedRowSize() to calculate buffer sizes. The difference is in calculating the number of rows, where what matters is the formula in BE. http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java File fe/src/main/java/org/apache/impala/planner/PlanNode.java: http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java@732 PS1, Line 732: getFixedLenRowSize_ += desc.getByteSize(); > Can this be zero? I don't know. Added check for division by zero protection. -- To view, visit http://gerrit.cloudera.org:8080/21797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f Gerrit-Change-Number: 21797 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Fri, 13 Sep 2024 16:21:57 +0000 Gerrit-HasComments: Yes
