github-actions[bot] commented on code in PR #69304: URL: https://github.com/apache/doris/pull/69304#discussion_r4579271307
########## be/src/exprs/aggregate/aggregate_function_legacy_nullable.h: ########## @@ -0,0 +1,141 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include <cstddef> +#include <cstdint> +#include <memory> +#include <utility> + +#include "exprs/aggregate/aggregate_function.h" + +namespace doris { + +template <typename T> +struct FunctionStableTransfer; + +// Published flag-plus-state String columns retain a separate codec, while every +// calculation delegates to V2. Keep the concrete function type so generic batch +// readers can inline create/decode/merge/destroy instead of making virtual calls +// for each state. Native readers retain their separate scratch capability. +template <typename Function> +class AggregateFunctionLegacyNullableState final + : public IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> { + using Base = IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>; + +public: + explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> function) + : Base(function->get_argument_types()), _function(std::move(function)) {} + + String get_name() const override { return _function->get_name(); } + IAggregateFunction* transmit_to_stable() override { + if (auto* stable = _function->transmit_to_stable()) { + using StableFunction = typename FunctionStableTransfer<Function>::FunctionStable; + return new AggregateFunctionLegacyNullableState<StableFunction>( + std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable))); + } + return nullptr; + } + DataTypePtr get_return_type() const override { return _function->get_return_type(); } + void set_version(int version_) override { + Base::set_version(version_); + _function->set_version(version_); + } + void set_query_context(QueryContext* ctx) override { _function->set_query_context(ctx); } + bool is_blockable() const override { return _function->is_blockable(); } Review Comment: [P2] Preserve window spill capabilities in the legacy adapter With the documented `be_exec_version=19` upgrade pin, new FE requests send the compatibility flag as false, so nullable SUM/AVG and numeric MIN/MAX windows are wrapped here. The adapter inherits `window_spill_strategy() = UNSUPPORTED` instead of forwarding the underlying `PARTITION_REDUCE` capability. `AnalyticSinkOperatorX::_prepare_spill` consequently disables spill for e.g. `SUM(nullable_v) OVER (PARTITION BY k)`, allowing a large partition to exceed its memory limit where the previous nullable wrapper could spill. Please forward the window spill capability methods and cover the legacy adapter in the spill tests. ########## be/src/exprs/aggregate/aggregate_function_legacy_nullable.h: ########## @@ -0,0 +1,141 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include <cstddef> +#include <cstdint> +#include <memory> +#include <utility> + +#include "exprs/aggregate/aggregate_function.h" + +namespace doris { + +template <typename T> +struct FunctionStableTransfer; + +// Published flag-plus-state String columns retain a separate codec, while every +// calculation delegates to V2. Keep the concrete function type so generic batch +// readers can inline create/decode/merge/destroy instead of making virtual calls +// for each state. Native readers retain their separate scratch capability. +template <typename Function> +class AggregateFunctionLegacyNullableState final + : public IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> { + using Base = IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>; + +public: + explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> function) + : Base(function->get_argument_types()), _function(std::move(function)) {} + + String get_name() const override { return _function->get_name(); } + IAggregateFunction* transmit_to_stable() override { + if (auto* stable = _function->transmit_to_stable()) { + using StableFunction = typename FunctionStableTransfer<Function>::FunctionStable; + return new AggregateFunctionLegacyNullableState<StableFunction>( + std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable))); + } + return nullptr; + } + DataTypePtr get_return_type() const override { return _function->get_return_type(); } + void set_version(int version_) override { + Base::set_version(version_); + _function->set_version(version_); + } + void set_query_context(QueryContext* ctx) override { _function->set_query_context(ctx); } + bool is_blockable() const override { return _function->is_blockable(); } + bool supported_incremental_mode() const override { + return _function->supported_incremental_mode(); + } + bool result_column_could_resize() const override { + return _function->result_column_could_resize(); + } + void execute_function_with_incremental(int64_t partition_start, int64_t partition_end, + int64_t frame_start, int64_t frame_end, + AggregateDataPtr place, const IColumn** columns, + Arena& arena, bool previous_is_nul, bool end_is_nul, + bool has_null, UInt8* use_null_result, + UInt8* could_use_previous_result) const override { + _function->execute_function_with_incremental( + partition_start, partition_end, frame_start, frame_end, place, columns, arena, + previous_is_nul, end_is_nul, has_null, use_null_result, could_use_previous_result); + } + void insert_result_into_range(ConstAggregateDataPtr place, IColumn& column, size_t start, + size_t end) const override { + _function->insert_result_into_range(place, column, start, end); + } + // Do not forward the native reader's scratch capability: this adapter reads String + // columns through generic readers even when the nested function reads native fixed states. + void create(AggregateDataPtr place) const override { _function->create(place); } + void destroy(AggregateDataPtr place) const noexcept override { _function->destroy(place); } + void reset(AggregateDataPtr place) const override { _function->reset(place); } + bool is_trivial() const override { return _function->is_trivial(); } + size_t size_of_data() const override { return _function->size_of_data(); } + size_t align_of_data() const override { return _function->align_of_data(); } + void check_input_columns_type(const IColumn** columns) const override { + _function->check_input_columns_type(columns); + } + void check_result_column_type(const IColumn& column) const override { + _function->check_result_column_type(column); + } + void add(AggregateDataPtr place, const IColumn** columns, ssize_t row, + Arena& arena) const override { + _function->add(place, columns, row, arena); + } + void streaming_agg_serialize_to_column(const IColumn** columns, MutableColumnPtr& dst, + size_t rows, Arena& arena) const override { + _function->streaming_agg_serialize_to_column_legacy(columns, dst, rows, arena); + } + void add_batch(size_t rows, AggregateDataPtr* places, size_t offset, const IColumn** columns, + Arena& arena, bool agg_many) const override { + _function->add_batch(rows, places, offset, columns, arena, agg_many); + } + void add_batch_selected(size_t rows, AggregateDataPtr* places, size_t offset, + const IColumn** columns, Arena& arena) const override { + _function->add_batch_selected(rows, places, offset, columns, arena); + } + void add_batch_single_place(size_t rows, AggregateDataPtr place, const IColumn** columns, + Arena& arena) const override { + _function->add_batch_single_place(rows, place, columns, arena); + } + void add_batch_range(size_t begin, size_t end, AggregateDataPtr place, const IColumn** columns, + Arena& arena, bool has_null) override { + _function->add_batch_range(begin, end, place, columns, arena, has_null); + } + void merge(AggregateDataPtr place, ConstAggregateDataPtr rhs, Arena& arena) const override { + _function->merge(place, rhs, arena); + } + void serialize(ConstAggregateDataPtr place, BufferWritable& buf) const override { + _function->serialize(place, buf); + } + void deserialize(AggregateDataPtr place, BufferReadable& buf, Arena& arena) const override { + _function->deserialize(place, buf, arena); + } + void deserialize_and_merge(AggregateDataPtr place, AggregateDataPtr rhs, BufferReadable& buf, + Arena& arena) const override { + _function->deserialize(rhs, buf, arena); Review Comment: [P2] Preserve direct DISTINCT merging in the byte codec This eagerly deserializes a fresh RHS state into the destination's long-lived arena before merging it. For nullable `multi_distinct_count(VARIANT)`, the former V1 path called the nested `deserialize_and_merge` specialization, which decodes into the destination and checks membership before copying a key. Here every repeated partial first copies its keys into the empty RHS; destroying that scratch hash set does not release those arena bytes. Merging N identical partials containing one K-byte VARIANT value therefore retains roughly (N+1)*K key bytes despite cardinality remaining 1, instead of roughly K. This affects pre-20 wire merging and the persisted nullable AGG_STATE adapter. Preserve nullable/marker handling while delegating valid payloads to the nested direct merge, and cover repeated duplicate partials with an arena-growth test. ########## be/src/exprs/aggregate/aggregate_function_null_v2.h: ########## @@ -236,67 +281,121 @@ class AggregateFunctionNullBaseInlineV2 : public IAggregateFunctionHelper<Derive nested_function->serialize_to_column(places, offset + prefix_size, nested_col_ptr, num_rows); } else { - nested_function->serialize_to_column(places, offset, dst, num_rows); + nested_function->serialize_to_column(places, offset + prefix_size, dst, num_rows); } } void streaming_agg_serialize_to_column(const IColumn** columns, MutableColumnPtr& dst, const size_t num_rows, Arena& arena) const override { - const auto* src_nullable_col = assert_cast<const ColumnNullable*>(columns[0]); - const auto* __restrict src_null_map_data = src_nullable_col->get_null_map_data().data(); + assert_cast<const Derived*>(this)->prepare_streaming_columns( + columns, num_rows, [&](const IColumn** nested_columns, const UInt8* null_map) { + streaming_serialize(nested_columns, null_map, dst, num_rows, arena); + }); + } - size_t nested_size = nested_function->size_of_data(); - std::vector<AggregateDataPtr> nested_places(num_rows); - std::vector<char> places_data(num_rows * nested_size); - for (size_t i = 0; i < num_rows; ++i) { - nested_places[i] = places_data.data() + i * nested_size; - } + template <typename Serialize> + void prepare_streaming_columns(const IColumn** columns, size_t num_rows, + Serialize&& serialize) const { + const auto& column = assert_cast<const ColumnNullable&>(*columns[0]); + const IColumn* nested_column = &column.get_nested_column(); + serialize(&nested_column, column.get_null_map_data().data()); + } - if (!nested_function->is_trivial()) { - for (int i = 0; i < num_rows; ++i) { - try { - nested_function->create(nested_places[i]); - } catch (...) { - for (int j = 0; j < i; ++j) { - nested_function->destroy(nested_places[j]); + void streaming_agg_serialize_to_column_legacy(const IColumn** columns, MutableColumnPtr& dst, + size_t num_rows, Arena&) const { + if (num_rows == 0) { + return; + } + // A hidden NULL array may shift later physical offsets, e.g. [NULL,[1]] vs [[2],[3]]. + // Normalize visible rows once for both native and historical String output. + assert_cast<const Derived*>(this)->prepare_streaming_columns( + columns, num_rows, [&](const IColumn** nested_columns, const UInt8* null_map) { + VectorBufferWriter writer(assert_cast<ColumnString&>(*dst)); + AlignedAggregateData storage(this->size_of_data(), this->align_of_data()); + auto* place = storage.data(); + Arena row_arena; + for (size_t row = 0; row < num_rows; ++row) { + row_arena.clear(); + create(place); + DEFER({ destroy(place); }); + if (!null_map[row]) { + set_flag(place); + nested_function->add(nested_place(place), nested_columns, row, + row_arena); + } + serialize(place, writer); + writer.commit(); } - throw; - } + }); + } + +protected: + void streaming_serialize(const IColumn** columns, const UInt8* null_map, MutableColumnPtr& dst, + size_t num_rows, Arena& arena) const { + if (num_rows == 0) { + return; + } + MutableColumnPtr nested_dst; + ColumnNullable* nullable_dst = nullptr; + if (native_state_is_nullable()) { + nullable_dst = &assert_cast<ColumnNullable&>(*dst); + nested_dst = nullable_dst->get_nested_column().assert_mutable(); + auto& dst_null_map = nullable_dst->get_null_map_data(); + dst_null_map.assign(null_map, null_map + num_rows); + } else { + nested_dst = std::move(dst); + } + DEFER({ + if (!native_state_is_nullable()) { + dst = std::move(nested_dst); } + }); + if constexpr (requires(const NestFunction& function) { + function.streaming_agg_serialize_to_column_with_null_map( + columns, nested_dst, num_rows, null_map); + }) { + nested_function->streaming_agg_serialize_to_column_with_null_map(columns, nested_dst, + num_rows, null_map); + return; } - Defer destroy_places = {[&]() { - if (!nested_function->is_trivial()) { - for (int i = 0; i < num_rows; ++i) { - nested_function->destroy(nested_places[i]); + const bool has_null = + std::any_of(null_map, null_map + num_rows, [](UInt8 value) { return value != 0; }); + if (!has_null) { + nested_function->streaming_agg_serialize_to_column(columns, nested_dst, num_rows, + arena); + return; + } + // A NULL row has no contribution (not even an empty bitmap intersection). + // Keep only one aligned state alive: wide/string/set states must not grow with batch size. + Arena state_arena; + Arena row_arena; + AggregateDataPtr place = nullptr; + for (size_t row = 0; row < num_rows; ++row) { + if (native_state_is_nullable()) { + if (null_map[row]) { + nested_dst->insert_default(); Review Comment: [P2] Count skipped NULL rows in the streaming writers For a profiled `ai_agg(resource, nullable_text, task)` with high-cardinality GROUP BY, the local streaming operator can switch to passthrough and reach this loop even at execution version 20 with statistics correctly attached. NULL rows now insert a default and bypass the wrapper's `record_null_input` hook; only valid rows reach the nested `add`. Consequently both `AIFunctionInputRows` and `AIFunctionSkippedNullRows` omit every passed-through NULL row, and downstream merges deliberately do not recount input. The legacy streaming loop has the same omission. Account skipped rows once per prepared batch in both writers, preserving the previous variadic accounting, and add mixed/all-NULL streaming statistics coverage. ########## be/src/exprs/aggregate/aggregate_function_legacy_nullable.h: ########## @@ -0,0 +1,141 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include <cstddef> +#include <cstdint> +#include <memory> +#include <utility> + +#include "exprs/aggregate/aggregate_function.h" + +namespace doris { + +template <typename T> +struct FunctionStableTransfer; + +// Published flag-plus-state String columns retain a separate codec, while every +// calculation delegates to V2. Keep the concrete function type so generic batch +// readers can inline create/decode/merge/destroy instead of making virtual calls +// for each state. Native readers retain their separate scratch capability. +template <typename Function> +class AggregateFunctionLegacyNullableState final + : public IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>> { + using Base = IAggregateFunctionHelper<AggregateFunctionLegacyNullableState<Function>>; + +public: + explicit AggregateFunctionLegacyNullableState(std::shared_ptr<Function> function) + : Base(function->get_argument_types()), _function(std::move(function)) {} + + String get_name() const override { return _function->get_name(); } + IAggregateFunction* transmit_to_stable() override { + if (auto* stable = _function->transmit_to_stable()) { + using StableFunction = typename FunctionStableTransfer<Function>::FunctionStable; + return new AggregateFunctionLegacyNullableState<StableFunction>( + std::shared_ptr<StableFunction>(assert_cast<StableFunction*>(stable))); + } + return nullptr; + } + DataTypePtr get_return_type() const override { return _function->get_return_type(); } + void set_version(int version_) override { + Base::set_version(version_); + _function->set_version(version_); + } + void set_query_context(QueryContext* ctx) override { _function->set_query_context(ctx); } Review Comment: [P2] Forward AI statistics through the legacy codec For nullable `ai_agg` at execution versions below 20, `AggFnEvaluator::prepare` installs this adapter (the aggregate has multiple arguments), then calls `set_ai_statistics`. This class forwards the query context but inherits the base no-op statistics setter, leaving the nested `AggregateFunctionAIAgg` executor without its statistics owner. With profiling enabled, input/skipped rows, requests, parsing and token usage therefore disappear from the query profile throughout the documented version-19 upgrade pin. Forward `set_ai_statistics` to `_function` and exercise the version-19 adapter in the AI statistics test; the changed test currently uses the raw factory function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
