github-actions[bot] commented on code in PR #70645:
URL: https://github.com/apache/doris/pull/70645#discussion_r4611571613
##########
be/src/exprs/function/array/function_arrays_overlap.cpp:
##########
@@ -21,6 +21,45 @@
namespace doris {
+Status FunctionArraysOverlap::open(FunctionContext* context,
+ FunctionContext::FunctionStateScope scope) {
+ if (scope != FunctionContext::FunctionStateScope::FRAGMENT_LOCAL) {
+ return Status::OK();
+ }
+ const bool left_const = context->is_col_constant(0);
+ const bool right_const = context->is_col_constant(1);
+ // Prepare only when exactly one argument is constant. With two constants,
+ // VectorizedFnCall caches the entire result; e.g. arrays_overlap([1, 2,
...], [])
+ // must not build a set before the empty-array check.
+ if (left_const == right_const) {
+ return Status::OK();
+ }
+ const int const_arg = left_const ? 0 : 1;
+
+ const auto& column = context->get_constant_col(const_arg)->column_ptr();
+ const auto& array_type =
+ assert_cast<const
DataTypeArray&>(*remove_nullable(context->get_arg_type(const_arg)));
+ const auto element_type = remove_nullable(array_type.get_nested_type());
+ auto prepare = [&](const auto& type) {
+ constexpr PrimitiveType PType = std::decay_t<decltype(type)>::PType;
+ const auto view = ColumnArrayView<PType>::create(column);
+ // A NULL array has no elements to prepare; execute propagates its
outer NULL.
+ if (!view.is_null_at(0)) {
Review Comment:
[P2] Bound the preparation cost for large constant arrays
Preparation hashes the entire constant before any row reaches the existing
NULL/empty fast paths. For `SELECT arrays_overlap(ints, array_range(1000000))
FROM t` with empty or NULL `ints`, the previous implementation performed no set
insertion, whereas this path inserts a million keys and retains the table even
though it is never probed. With a short input containing `[0]`, the previous
smaller-array path inserted one key and matched the first range element
immediately, so small scans also regress sharply. This extra work remains even
after fixing memory accounting. Preserve a fallback for large constants or
defer/bound preparation so it can pay back its setup cost, and include `open()`
plus large-constant/short-or-empty inputs in the performance coverage; the new
benchmark excludes both.
##########
be/src/exprs/function/array/function_arrays_overlap.cpp:
##########
@@ -21,6 +21,45 @@
namespace doris {
+Status FunctionArraysOverlap::open(FunctionContext* context,
+ FunctionContext::FunctionStateScope scope) {
+ if (scope != FunctionContext::FunctionStateScope::FRAGMENT_LOCAL) {
+ return Status::OK();
+ }
+ const bool left_const = context->is_col_constant(0);
+ const bool right_const = context->is_col_constant(1);
+ // Prepare only when exactly one argument is constant. With two constants,
+ // VectorizedFnCall caches the entire result; e.g. arrays_overlap([1, 2,
...], [])
+ // must not build a set before the empty-array check.
+ if (left_const == right_const) {
+ return Status::OK();
+ }
+ const int const_arg = left_const ? 0 : 1;
+
+ const auto& column = context->get_constant_col(const_arg)->column_ptr();
+ const auto& array_type =
+ assert_cast<const
DataTypeArray&>(*remove_nullable(context->get_arg_type(const_arg)));
+ const auto element_type = remove_nullable(array_type.get_nested_type());
+ auto prepare = [&](const auto& type) {
+ constexpr PrimitiveType PType = std::decay_t<decltype(type)>::PType;
+ const auto view = ColumnArrayView<PType>::create(column);
+ // A NULL array has no elements to prepare; execute propagates its
outer NULL.
+ if (!view.is_null_at(0)) {
+ auto prepared = std::make_shared<PreparedSet<PType>>(const_arg);
+ prepared->set.insert_array(view[0]);
Review Comment:
[P1] Account for the memory retained by the prepared set
This new fragment-local table uses `OverlapSetImpl`'s `phmap::flat_hash_set`
aliases with their default `std::allocator`; neither `PreparedSet` nor `open()`
accounts its capacity. The malloc hooks only forward to jemalloc, so this
storage bypasses Doris's memory-limit checks and tracker consumption. For
`arrays_overlap(array_col, array_range(1000000))`, a nonconstant column now
causes a million-key table to remain allocated for the fragment, even when
every row is empty (the previous path allocated no table for those rows).
Tracking the constant column does not cover this additional allocation, and
concurrent fragments can retain substantial uncharged memory. Use
allocator-aware storage, preserving the current equality semantics, or
explicitly check/account/release the retained capacity against the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]