Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21557 )
Change subject: IMPALA-13088, IMPALA-13109: Use RoaringBitmap instead of sorted vector of int64s ...................................................................... Patch Set 8: (3 comments) looks great! few questions, mainly related to memory management http://gerrit.cloudera.org:8080/#/c/21557/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21557/8//COMMIT_MSG@22 PS8, Line 22: lots of records It could be interesting how the deleted rows were selected, as roaring bitmap stores continuous runs in optimized way. http://gerrit.cloudera.org:8080/#/c/21557/8//COMMIT_MSG@41 PS8, Line 41: 10 executors cluster with sorted vectors failed to run the query because : executors crashed due to out-of-memory. Can you add a bit more info on this? What should be clarified is whether there are known issues with IcebergDeleteNode that can lead to OOM crashes. If I understand the memory management correctly then we have a fair estimation during planning about the memory consumption and set this as minimal reservation. If the cluster doesn't have that much memory then it shouldn't start running the query. Do you know what goes wrong? I don't mean to fix existing memory issues in this commit, but there should be a Jira about this. http://gerrit.cloudera.org:8080/#/c/21557/8//COMMIT_MSG@44 PS8, Line 44: Memory usage (VmRSS) for 10 executors: The memory estimations are not changed - this is intentional, as we assume that in the worst case the roaring bitmap will still need 8 byte per position, right? https://github.com/apache/impala/blob/00d0b0dda1e215d8e91ff52688fe6654bee52282/fe/src/main/java/org/apache/impala/planner/IcebergDeleteNode.java#L200 -- To view, visit http://gerrit.cloudera.org:8080/21557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib769965d094149e99c43e0044914d9ecccc76107 Gerrit-Change-Number: 21557 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Mon, 08 Jul 2024 18:13:54 +0000 Gerrit-HasComments: Yes
