siddharthteotia commented on code in PR #11200:
URL: https://github.com/apache/pinot/pull/11200#discussion_r1284932374
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/BaseDistinctAggregateAggregationFunction.java:
##########
@@ -112,6 +113,9 @@ public Set merge(Set intermediateResult1, Set
intermediateResult2) {
if (intermediateResult2.isEmpty()) {
return intermediateResult1;
}
+
+ Tracing.ThreadAccountantOps.sampleAndCheckInterruption();
Review Comment:
I am wondering if this will actually prevent the problem we saw happening
internally.
IIUC - Until this point in the code, merge hasn't consumed any memory. It's
the addAll() and resulting rehash etc that can blow up memory ?
Here is the exception from the heap dump.
```
2. Thread throwing OutOfMemoryError. found [What's
this?](https://aka.ms/jxray/#thread_throwing_oom)
This dump was created after OutOfMemoryError in the following thread:
Thread name: "brw-14", daemon: false
java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
it.unimi.dsi.fastutil.longs.LongOpenHashSet.rehash(LongOpenHashSet.java:592)
Local variables
it.unimi.dsi.fastutil.longs.LongOpenHashSet.tryCapacity(LongOpenHashSet.java:266)
it.unimi.dsi.fastutil.longs.LongOpenHashSet.addAll(LongOpenHashSet.java:283)
Local variables
org.apache.pinot.core.query.aggregation.function.BaseDistinctAggregateAggregationFunction.merge(BaseDistinctAggregateAggregationFunction.java:115)
Local variables
org.apache.pinot.core.query.aggregation.function.BaseDistinctAggregateAggregationFunction.merge(BaseDistinctAggregateAggregationFunction.java:47)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
