Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22062 )
Change subject: IMPALA-13487: Add profile counters for memory allocation ...................................................................... Patch Set 20: (1 comment) http://gerrit.cloudera.org:8080/#/c/22062/20//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22062/20//COMMIT_MSG@12 PS20, Line 12: The following counters are added: > Could this also counter Union materialization? I don't think that's covered No, scratch batch is only used in parquet/orc scanners. Parquet data-page-pool is only used in the parquet scanner. UnionNode doesn't reset the output RowBatch so RowBatchMemPoolFree* counters are not updated for it. However, I see UnionNode::GetNextMaterialized() allocates buffer for the output RowBatch: https://github.com/apache/impala/blob/0ed4e869de43532c72dab514850944c3a6036bd1/be/src/exec/union-node.cc#L217 I can add RowBatchMemPoolAllocDuration and RowBatchMemPoolAllocBytes so that will be tracked. UnionNode::GetNextMaterialized() also reset the child_batch which is not tracked by these counters: https://github.com/apache/impala/blob/0ed4e869de43532c72dab514850944c3a6036bd1/be/src/exec/union-node.cc#L245 There might be other places in UnionNode that have significant memory operations. We can add some counters like ChildBatchMemPoolFreeDuration in another patch, after analyzing all the memory allocation&free patterns of UnionNode. -- To view, visit http://gerrit.cloudera.org:8080/22062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I982315d96e6de20a3616f3bd2a2b4866d1ff4710 Gerrit-Change-Number: 22062 Gerrit-PatchSet: 20 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Xuebin Su <x...@cloudera.com> Gerrit-Comment-Date: Sat, 12 Apr 2025 03:20:54 +0000 Gerrit-HasComments: Yes
