github-actions[bot] commented on code in PR #70645:
URL: https://github.com/apache/doris/pull/70645#discussion_r4611571613


##########
be/src/exprs/function/array/function_arrays_overlap.cpp:
##########
@@ -21,6 +21,45 @@
 
 namespace doris {
 
+Status FunctionArraysOverlap::open(FunctionContext* context,
+                                   FunctionContext::FunctionStateScope scope) {
+    if (scope != FunctionContext::FunctionStateScope::FRAGMENT_LOCAL) {
+        return Status::OK();
+    }
+    const bool left_const = context->is_col_constant(0);
+    const bool right_const = context->is_col_constant(1);
+    // Prepare only when exactly one argument is constant. With two constants,
+    // VectorizedFnCall caches the entire result; e.g. arrays_overlap([1, 2, 
...], [])
+    // must not build a set before the empty-array check.
+    if (left_const == right_const) {
+        return Status::OK();
+    }
+    const int const_arg = left_const ? 0 : 1;
+
+    const auto& column = context->get_constant_col(const_arg)->column_ptr();
+    const auto& array_type =
+            assert_cast<const 
DataTypeArray&>(*remove_nullable(context->get_arg_type(const_arg)));
+    const auto element_type = remove_nullable(array_type.get_nested_type());
+    auto prepare = [&](const auto& type) {
+        constexpr PrimitiveType PType = std::decay_t<decltype(type)>::PType;
+        const auto view = ColumnArrayView<PType>::create(column);
+        // A NULL array has no elements to prepare; execute propagates its 
outer NULL.
+        if (!view.is_null_at(0)) {

Review Comment:
   [P2] Bound the preparation cost for large constant arrays
   
   Preparation hashes the entire constant before any row reaches the existing 
NULL/empty fast paths. For `SELECT arrays_overlap(ints, array_range(1000000)) 
FROM t` with empty or NULL `ints`, the previous implementation performed no set 
insertion, whereas this path inserts a million keys and retains the table even 
though it is never probed. With a short input containing `[0]`, the previous 
smaller-array path inserted one key and matched the first range element 
immediately, so small scans also regress sharply. This extra work remains even 
after fixing memory accounting. Preserve a fallback for large constants or 
defer/bound preparation so it can pay back its setup cost, and include `open()` 
plus large-constant/short-or-empty inputs in the performance coverage; the new 
benchmark excludes both.



##########
be/src/exprs/function/array/function_arrays_overlap.cpp:
##########
@@ -21,6 +21,45 @@
 
 namespace doris {
 
+Status FunctionArraysOverlap::open(FunctionContext* context,
+                                   FunctionContext::FunctionStateScope scope) {
+    if (scope != FunctionContext::FunctionStateScope::FRAGMENT_LOCAL) {
+        return Status::OK();
+    }
+    const bool left_const = context->is_col_constant(0);
+    const bool right_const = context->is_col_constant(1);
+    // Prepare only when exactly one argument is constant. With two constants,
+    // VectorizedFnCall caches the entire result; e.g. arrays_overlap([1, 2, 
...], [])
+    // must not build a set before the empty-array check.
+    if (left_const == right_const) {
+        return Status::OK();
+    }
+    const int const_arg = left_const ? 0 : 1;
+
+    const auto& column = context->get_constant_col(const_arg)->column_ptr();
+    const auto& array_type =
+            assert_cast<const 
DataTypeArray&>(*remove_nullable(context->get_arg_type(const_arg)));
+    const auto element_type = remove_nullable(array_type.get_nested_type());
+    auto prepare = [&](const auto& type) {
+        constexpr PrimitiveType PType = std::decay_t<decltype(type)>::PType;
+        const auto view = ColumnArrayView<PType>::create(column);
+        // A NULL array has no elements to prepare; execute propagates its 
outer NULL.
+        if (!view.is_null_at(0)) {
+            auto prepared = std::make_shared<PreparedSet<PType>>(const_arg);
+            prepared->set.insert_array(view[0]);

Review Comment:
   [P1] Account for the memory retained by the prepared set
   
   This new fragment-local table uses `OverlapSetImpl`'s `phmap::flat_hash_set` 
aliases with their default `std::allocator`; neither `PreparedSet` nor `open()` 
accounts its capacity. The malloc hooks only forward to jemalloc, so this 
storage bypasses Doris's memory-limit checks and tracker consumption. For 
`arrays_overlap(array_col, array_range(1000000))`, a nonconstant column now 
causes a million-key table to remain allocated for the fragment, even when 
every row is empty (the previous path allocated no table for those rows). 
Tracking the constant column does not cover this additional allocation, and 
concurrent fragments can retain substantial uncharged memory. Use 
allocator-aware storage, preserving the current equality semantics, or 
explicitly check/account/release the retained capacity against the query.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to