This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 489eecba70c [refactor](exec) Extract COW-safe tablet column validation
(#70562)
489eecba70c is described below
commit 489eecba70cba635e077666bbb79a0dc41edd976
Author: Mryange <[email protected]>
AuthorDate: Wed Nov 18 15:12:49 2026 +0800
[refactor](exec) Extract COW-safe tablet column validation (#70562)
### Background
Tablet write validation can truncate strings, round legacy decimals, and
deduplicate map keys. Expression outputs can share column trees, so
validation must preserve copy-on-write ownership before changing them.
### What problem does this PR solve?
Issue Number: None
Related PR: #70421 (superseded)
PR Type: Refactor
Problem Summary:
Tablet validation is embedded in the block converter as a large
recursive function that mixes type dispatch, nullable handling, row
filtering, and mutation. Extract it into a standalone, testable column
validator with per-type helpers and a shared validation context. The new
entry reads an immutable input tree and returns a validated column.
Scalar checks reuse input storage; complex types build new shells around
validated children, avoiding shared-column corruption and full-buffer
copies for read-only validation.
### Details
- **Change map:** Roughly 500 lines of embedded validation are replaced
by a standalone implementation and interface; the converter keeps block
orchestration. Add ownership/validation UTs and a 55-line regression
with generated output. No new column mutation accessor is needed.
- **Moves vs. logic changes:** Move scalar checks and decimal bound
caches into per-type helpers/context. Centralize physical nullable
handling and original-row mapping. Replace const-cast mutation with
immutable recursive traversal and fresh container shells. String
truncation returns a replacement; DecimalV2 clones only if rounding is
needed. Each new map shell deduplicates once, detaching child data only
when filtering duplicates.
- **Behavior change:** Validation no longer modifies shared source
columns or retains stale views after nested string replacement.
Inherited NULLs skip nested payload checks and rejected elements filter
their original outer row.
- **Compatibility:** No wire or storage format changes. DecimalV2 keeps
historical execution precision/scale (27, 9); enforcing declared schema
parameters and resolving map-key collisions after string truncation
remain outside this change. This refactoring targets master without
release pick requests.
- **Performance impact:** Read-only scalar columns and unchanged leaf
buffers remain shared, including offsets and null maps. Complex types
allocate new shells even when their children are unchanged; this trades
small container allocations for simple ownership without recursive
detachment. UTs verify storage reuse and source isolation. No throughput
benchmark was run.
- **Rollback:** Revert this commit; no metadata or data migration is
needed.
### Testing
- `BUILD_TYPE_UT=ASAN sh run-be-ut.sh --run
--filter='TabletColumnValidatorTest.*' -j48`: 24 tests passed on the
development machine. Covers shared read-only leaves, nested shell
construction, replacement/source isolation, map deduplication, NULL
propagation and decimal compatibility.
- Repository clang-format 16 scripts, build hygiene, `git diff --check`,
and prescribed clang-tidy on the changed source TUs and validator UT:
passed.
- No standalone BE build or end-to-end regression was rerun for this
revision. The ASAN UT script compiled the affected production code and
test binary; the existing shared-map regression and generated output are
retained.
### Release note
Fix incorrect results or failures when inserting shared complex columns
that require tablet validation.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: focused ASAN UTs and static
checks passed; standalone BE build and end-to-end regression not rerun
after this change.
### AI Disclosure
- Tool: Codex (GPT-6).
- Generated: Code, tests, and this description.
- Human review: The author reviewed the design and an earlier source
version; the final changes remain available for review.
---
be/src/exec/sink/tablet_column_validator.cpp | 475 +++++++++++++++++++
be/src/exec/sink/tablet_column_validator.h | 61 +++
be/src/exec/sink/vtablet_block_convertor.cpp | 497 +-------------------
be/src/exec/sink/vtablet_block_convertor.h | 41 +-
be/test/exec/sink/tablet_column_validator_test.cpp | 511 +++++++++++++++++++++
.../map/test_map_insert_shared_column.out | 4 +
.../map/test_map_insert_shared_column.groovy | 55 +++
7 files changed, 1127 insertions(+), 517 deletions(-)
diff --git a/be/src/exec/sink/tablet_column_validator.cpp
b/be/src/exec/sink/tablet_column_validator.cpp
new file mode 100644
index 00000000000..0d0b7c049cf
--- /dev/null
+++ b/be/src/exec/sink/tablet_column_validator.cpp
@@ -0,0 +1,475 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "exec/sink/tablet_column_validator.h"
+
+#include <algorithm>
+#include <string>
+#include <type_traits>
+#include <utility>
+
+#include "common/config.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/binary_cast.hpp"
+#include "core/block/block.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "core/wide_integer_to_string.h"
+#include "exprs/function/function_helpers.h"
+#include "exprs/function/simple_function_factory.h"
+#include "runtime/runtime_state.h"
+#include "storage/olap_common.h"
+#include "util/defer_op.h"
+
+namespace doris {
+
+Status TabletValidateContext::reject(size_t row, const fmt::memory_buffer&
message) const {
+ filter_map[row] = true;
+ return state->append_error_msg_to_file(
+ []() -> std::string { return ""; },
+ [&]() { return fmt::to_string(error_prefix) +
fmt::to_string(message); });
+}
+
+namespace {
+
+template <bool is_min>
+DecimalV2Value get_decimalv2_min_or_max(TabletValidateContext& context, const
DataTypePtr& type) {
+ auto& cache = is_min ? context.min_decimalv2_val :
context.max_decimalv2_val;
+ const auto precision = type->get_precision();
+ const auto scale = type->get_scale();
+ const auto key = std::pair {precision, scale};
+ auto iter = cache.find(key);
+ if (iter != cache.end()) {
+ return iter->second;
+ }
+ DecimalV2Value value;
+ if constexpr (is_min) {
+ value.to_min_decimal(precision, scale);
+ } else {
+ value.to_max_decimal(precision, scale);
+ }
+ cache.emplace(key, value);
+ return value;
+}
+
+template <typename DecimalType, bool IsMin>
+DecimalType get_decimalv3_min_or_max(TabletValidateContext& context, const
DataTypePtr& type) {
+ std::map<int, typename DecimalType::NativeType>* pmap;
+ if constexpr (std::is_same_v<DecimalType, Decimal32>) {
+ pmap = IsMin ? &context.min_decimal32_val : &context.max_decimal32_val;
+ } else if constexpr (std::is_same_v<DecimalType, Decimal64>) {
+ pmap = IsMin ? &context.min_decimal64_val : &context.max_decimal64_val;
+ } else if constexpr (std::is_same_v<DecimalType, Decimal128V3>) {
+ pmap = IsMin ? &context.min_decimal128_val :
&context.max_decimal128_val;
+ } else {
+ pmap = IsMin ? &context.min_decimal256_val :
&context.max_decimal256_val;
+ }
+
+ // found
+ auto iter = pmap->find(type->get_precision());
+ if (iter != pmap->end()) {
+ return DecimalType(iter->second);
+ }
+
+ DecimalType value;
+ if constexpr (IsMin) {
+ value = min_decimal_value<DecimalType::PType>(type->get_precision());
+ } else {
+ value = max_decimal_value<DecimalType::PType>(type->get_precision());
+ }
+ pmap->emplace(type->get_precision(), value.value);
+ return value;
+}
+
+// Borrowed views: local positions map to rows in the original output block.
+// null_map includes NULLs inherited from enclosing nullable containers.
+struct ValidationRows {
+ const IColumn::Permutation* rows = nullptr;
+ const UInt8* null_map = nullptr;
+
+ size_t original_row(size_t i) const { return rows ? (*rows)[i] : i; }
+
+ bool should_validate(size_t i, const TabletValidateContext& context) const
{
+ return !context.filter_map[original_row(i)] && (!null_map ||
!null_map[i]);
+ }
+};
+
+Result<ColumnPtr> validate_column(const ColumnPtr& column, const DataTypePtr&
type,
+ TabletValidateContext& context, const
ValidationRows& rows);
+
+Result<ColumnPtr> validate_string(ColumnPtr column, const DataTypePtr& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ int limit = config::string_type_length_soft_limit_bytes;
+ int len = -1;
+ const auto* type_str = check_and_get_data_type<DataTypeString>(type.get());
+ if (type_str && type_str->len() >= 0) {
+ len = type_str->len();
+ limit = std::min(limit, len);
+ }
+
+ const auto& strings = assert_cast<const ColumnString&>(*column);
+ const auto& offsets = strings.get_offsets();
+ size_t invalid_count = 0;
+ for (size_t i = 0; i < column->size(); ++i) {
+ invalid_count += rows.should_validate(i, context) && offsets[i] -
offsets[i - 1] > limit;
+ }
+ if (!invalid_count) {
+ return column;
+ }
+
+ // Non-strict loads truncate in characters, but the tablet limit is in
bytes.
+ // For example, two Chinese characters can still exceed a VARCHAR(2) byte
limit.
+ if (type_str && !context.state->enable_insert_strict()) {
+ auto int_type =
+
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_INT, 0,
0);
+ ColumnsWithTypeAndName arguments {{nullptr, type, "string column"},
+ {nullptr, int_type, "pos"},
+ {nullptr, int_type, "len"}};
+ auto function = SimpleFunctionFactory::instance().get_function(
+ "substring", arguments, type, {},
context.state->be_exec_version());
+ if (!function) {
+ return ResultError(Status::InternalError("get function substring
failed"));
+ }
+ ColumnPtr result;
+ {
+ const auto size = column->size();
+ auto pos = int_type->create_column_const(size,
to_field<TYPE_INT>(1));
+ auto length = int_type->create_column_const(size,
to_field<TYPE_INT>(limit));
+ Block block({{column->get_ptr(), type, "string column"},
+ {pos, int_type, "pos"},
+ {length, int_type, "len"},
+ {nullptr, type, "result"}});
+ RETURN_IF_ERROR_RESULT(function->execute(nullptr, block, {0, 1,
2}, 3, size));
+ result = std::move(block.get_by_position(3).column);
+ }
+ column = std::move(result);
+ }
+
+ const auto& checked = assert_cast<const ColumnString&>(*column);
+ for (size_t i = 0; i < column->size(); ++i) {
+ if (!rows.should_validate(i, context)) {
+ continue;
+ }
+ auto value = checked.get_data_at(i);
+ if (std::cmp_greater(value.size, static_cast<size_t>(limit))) {
+ fmt::memory_buffer message;
+ if (std::cmp_greater(value.size, static_cast<size_t>(len))) {
+ fmt::format_to(message, "the length of input is too long than
schema. ");
+ fmt::format_to(message, "first 32 bytes of input str: [{}] ",
value.to_prefix(32));
+ fmt::format_to(message, "schema length: {}; actual length: {};
", len, value.size);
+ } else {
+ fmt::format_to(message, "the length of input string is too
long than vec schema. ");
+ fmt::format_to(message, "first 32 bytes of input str: [{}] ",
value.to_prefix(32));
+ fmt::format_to(message, "schema length: {}; limit length: {};
actual length: {}; ",
+ len, limit, value.size);
+ }
+ RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i),
message));
+ }
+ }
+ return column;
+}
+
+Status validate_jsonb(const ColumnString& column, TabletValidateContext&
context,
+ const ValidationRows& rows) {
+ for (size_t i = 0; i < column.size(); ++i) {
+ if (rows.should_validate(i, context) && column.get_data_at(i).size ==
0) {
+ fmt::memory_buffer message;
+ fmt::format_to(message, "jsonb with size 0 is invalid");
+ RETURN_IF_ERROR(context.reject(rows.original_row(i), message));
+ }
+ }
+ return Status::OK();
+}
+
+Result<ColumnPtr> validate_decimal_v2(const ColumnPtr& input, const
DataTypePtr& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ // DecimalV2 is deprecated. Preserve its historical validation using
execution
+ // precision/scale (27, 9), rather than the original schema parameters. For
+ // example, DECIMAL(9, 2) still validates 1.235 at scale 9 here; enforcing
the
+ // declared scale and range is outside this COW refactoring.
+ const auto& column = assert_cast<const ColumnDecimal128V2&>(*input);
+ MutableColumnPtr rewritten;
+ const auto precision = type->get_precision();
+ const auto scale = type->get_scale();
+ const auto max = get_decimalv2_min_or_max<false>(context, type);
+ const auto min = get_decimalv2_min_or_max<true>(context, type);
+ for (size_t i = 0; i < column.size(); ++i) {
+ if (!rows.should_validate(i, context)) {
+ continue;
+ }
+ auto value = binary_cast<Int128, DecimalV2Value>(column.get_data()[i]);
+ fmt::memory_buffer message;
+ if (value.greater_than_scale(scale)) {
+ auto code = value.round(&value, scale, HALF_UP);
+ if (!rewritten) {
+ rewritten = column.clone();
+ }
+ assert_cast<ColumnDecimal128V2&>(*rewritten).get_data()[i] = value;
+ if (code != E_DEC_OK) {
+ fmt::format_to(message, "round one decimal failed.value={}; ",
value.to_string());
+ }
+ }
+ if (value > max || value < min) {
+ fmt::format_to(message,
+ "decimal value is not valid for definition,
value={}, "
+ "precision={}, scale={}, min={}, max={}; ",
+ value.to_string(), precision, scale,
min.to_string(), max.to_string());
+ }
+ if (message.size()) {
+ RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i),
message));
+ }
+ }
+ return rewritten ? ColumnPtr(std::move(rewritten)) : input;
+}
+
+// DecimalV3 needs only a range check; use one template instead of four macro
bodies.
+template <typename DecimalType>
+Status validate_decimal_v3(const ColumnDecimal<DecimalType::PType>& column,
const DataTypePtr& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ const auto max = get_decimalv3_min_or_max<DecimalType, false>(context,
type);
+ const auto min = get_decimalv3_min_or_max<DecimalType, true>(context,
type);
+ size_t invalid_count = 0;
+ for (const auto value : column.get_data()) {
+ invalid_count += value > max || value < min;
+ }
+ if (!invalid_count) {
+ return Status::OK();
+ }
+ for (size_t i = 0; i < column.size(); ++i) {
+ if (!rows.should_validate(i, context)) {
+ continue;
+ }
+ const auto value = column.get_data()[i];
+ if (value > max || value < min) {
+ fmt::memory_buffer message;
+ fmt::format_to(message,
+ "decimal value is not valid for definition,
value={}, "
+ "precision={}, scale={}, min={}, max={}; ",
+ value.value, type->get_precision(),
type->get_scale(), min.value,
+ max.value);
+ RETURN_IF_ERROR(context.reject(rows.original_row(i), message));
+ }
+ }
+ return Status::OK();
+}
+
+// For [NULL, [x, y]], both x and y map to outer row 1; elements underneath a
+// NULL parent are skipped even if their physical payload happens to be
invalid.
+struct NestedRows {
+ IColumn::Permutation rows;
+ NullMap null_map;
+
+ NestedRows(const ColumnArray::Offsets64& offsets, const ValidationRows&
parent)
+ : rows(offsets.back()) {
+ if (parent.null_map) {
+ null_map.resize(rows.size());
+ }
+ for (size_t i = 0; i < offsets.size(); ++i) {
+ for (size_t j = offsets[i - 1]; j < offsets[i]; ++j) {
+ rows[j] = parent.original_row(i);
+ if (parent.null_map) {
+ null_map[j] = parent.null_map[i];
+ }
+ }
+ }
+ }
+
+ ValidationRows view() const {
+ return {.rows = &rows, .null_map = null_map.empty() ? nullptr :
null_map.data()};
+ }
+};
+
+Result<ColumnPtr> validate_array(const ColumnArray& column, const
DataTypeArray& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ NestedRows nested(column.get_offsets(), rows);
+ const auto prefix_size = context.error_prefix.size();
+ Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+ fmt::format_to(context.error_prefix, "ARRAY type failed: ");
+ auto data =
+ validate_column(column.get_data_ptr(), type.get_nested_type(),
context, nested.view());
+ if (!data) {
+ return unexpected(std::move(data.error()));
+ }
+ return ColumnPtr(ColumnArray::create(data.value(),
column.get_offsets_ptr()));
+}
+
+Result<ColumnPtr> validate_map(const ColumnMap& input, const DataTypeMap& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ // Deduplication changes only this new shell. The existing implementation
+ // detaches child data only when duplicate keys require filtering.
+ auto result = ColumnMap::create(input.get_keys_ptr(),
input.get_values_ptr(),
+ input.get_offsets_ptr());
+ RETURN_IF_ERROR_RESULT(result->deduplicate_keys(false));
+ const auto& column = *result;
+ NestedRows nested(column.get_offsets(), rows);
+ const auto prefix_size = context.error_prefix.size();
+ Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+ fmt::format_to(context.error_prefix, "MAP type failed: ");
+ auto keys = validate_column(column.get_keys_ptr(), type.get_key_type(),
context, nested.view());
+ if (!keys) {
+ return unexpected(std::move(keys.error()));
+ }
+ auto values =
+ validate_column(column.get_values_ptr(), type.get_value_type(),
context, nested.view());
+ if (!values) {
+ return unexpected(std::move(values.error()));
+ }
+ result->get_keys_ptr() = std::move(keys.value());
+ result->get_values_ptr() = std::move(values.value());
+ return ColumnPtr(std::move(result));
+}
+
+Result<ColumnPtr> validate_struct(const ColumnStruct& column, const
DataTypeStruct& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ DCHECK_EQ(type.get_elements().size(), column.tuple_size());
+ const auto prefix_size = context.error_prefix.size();
+ Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+ fmt::format_to(context.error_prefix, "STRUCT type failed: ");
+ Columns children;
+ children.reserve(column.tuple_size());
+ for (size_t i = 0; i < column.tuple_size(); ++i) {
+ auto child = validate_column(column.get_column_ptr(i),
type.get_element(i), context, rows);
+ if (!child) {
+ return unexpected(std::move(child.error()));
+ }
+ children.push_back(std::move(child.value()));
+ }
+ return ColumnPtr(ColumnStruct::create(std::move(children)));
+}
+
+Result<ColumnPtr> validate_nullable(const ColumnNullable& column, const
DataTypePtr& type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ const auto& null_map = column.get_null_map_data();
+ // A physically nullable column can target NOT NULL (including BITMAP).
NULLs
+ // beneath a NULL ancestor do not represent values being written to that
target.
+ if (!type->is_nullable() || type->get_primitive_type() == TYPE_BITMAP) {
+ for (size_t i = 0; i < column.size(); ++i) {
+ if (null_map[i] && rows.should_validate(i, context)) {
+ fmt::memory_buffer message;
+ fmt::format_to(message, "null value for not null column,
type={}",
+ type->get_name());
+ RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i),
message));
+ }
+ }
+ }
+ NullMap combined;
+ const UInt8* effective_null_map = null_map.data();
+ if (rows.null_map) {
+ combined.resize(column.size());
+ for (size_t i = 0; i < column.size(); ++i) {
+ combined[i] = null_map[i] | rows.null_map[i];
+ }
+ effective_null_map = combined.data();
+ }
+ auto nested = validate_column(column.get_nested_column_ptr(),
remove_nullable(type), context,
+ {.rows = rows.rows, .null_map =
effective_null_map});
+ if (!nested) {
+ return unexpected(std::move(nested.error()));
+ }
+ return ColumnPtr(ColumnNullable::create(nested.value(),
column.get_null_map_column_ptr()));
+}
+
+Result<ColumnPtr> validate_column(const ColumnPtr& column, const DataTypePtr&
type,
+ TabletValidateContext& context, const
ValidationRows& rows) {
+ DCHECK(!rows.rows || rows.rows->size() == column->size());
+ const auto& source = *column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(source)) {
+ return validate_nullable(*nullable, type, context, rows);
+ }
+ // A non-nullable input can target Nullable(T); physical nullability is
+ // handled above, while dispatch must always use the unwrapped target type.
+ const auto nested_type = remove_nullable(type);
+ switch (nested_type->get_primitive_type()) {
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING:
+ return validate_string(column, nested_type, context, rows);
+ case TYPE_JSONB:
+ RETURN_IF_ERROR_RESULT(
+ validate_jsonb(assert_cast<const ColumnString&>(source),
context, rows));
+ return column;
+ case TYPE_DECIMALV2:
+ return validate_decimal_v2(column, nested_type, context, rows);
+ case TYPE_DECIMAL32:
+ RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal32>(
+ assert_cast<const ColumnDecimal32&>(source), nested_type,
context, rows));
+ return column;
+ case TYPE_DECIMAL64:
+ RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal64>(
+ assert_cast<const ColumnDecimal64&>(source), nested_type,
context, rows));
+ return column;
+ case TYPE_DECIMAL128I:
+ RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal128V3>(
+ assert_cast<const ColumnDecimal128V3&>(source), nested_type,
context, rows));
+ return column;
+ case TYPE_DECIMAL256:
+ RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal256>(
+ assert_cast<const ColumnDecimal256&>(source), nested_type,
context, rows));
+ return column;
+ case TYPE_ARRAY:
+ return validate_array(assert_cast<const ColumnArray&>(source),
+ assert_cast<const DataTypeArray&>(*nested_type),
context, rows);
+ case TYPE_MAP:
+ return validate_map(assert_cast<const ColumnMap&>(source),
+ assert_cast<const DataTypeMap&>(*nested_type),
context, rows);
+ case TYPE_STRUCT:
+ return validate_struct(assert_cast<const ColumnStruct&>(source),
+ assert_cast<const
DataTypeStruct&>(*nested_type), context, rows);
+ case TYPE_AGG_STATE:
+ // Some aggregate states are serialized into a ColumnString.
+ if (check_and_get_column<ColumnString>(source)) {
+ return validate_string(column, nested_type, context, rows);
+ }
+ return column;
+ default:
+ return column;
+ }
+}
+
+} // namespace
+
+Result<ColumnPtr> tablet_validate_column(const ColumnPtr& column, const
DataTypePtr& type,
+ TabletValidateContext& context) {
+ ColumnPtr result;
+ auto validate = [&]() -> Status {
+ RETURN_IF_CATCH_EXCEPTION({
+ auto checked = validate_column(column, type, context, {});
+ if (!checked) {
+ return std::move(checked.error());
+ }
+ result = std::move(checked.value());
+ return Status::OK();
+ });
+ };
+ RETURN_IF_ERROR_RESULT(validate());
+ return result;
+}
+
+} // namespace doris
diff --git a/be/src/exec/sink/tablet_column_validator.h
b/be/src/exec/sink/tablet_column_validator.h
new file mode 100644
index 00000000000..093fa1ca9f8
--- /dev/null
+++ b/be/src/exec/sink/tablet_column_validator.h
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+
+#include <fmt/format.h>
+
+#include <map>
+#include <vector>
+
+#include "common/status.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+
+namespace doris {
+
+class RuntimeState;
+
+// State shared by recursive tablet validation of one output block.
+struct TabletValidateContext {
+ explicit TabletValidateContext(std::vector<char>& filter_map) :
filter_map(filter_map) {}
+
+ Status reject(size_t row, const fmt::memory_buffer& message) const;
+
+ RuntimeState* state = nullptr;
+ fmt::memory_buffer error_prefix;
+ std::vector<char>& filter_map;
+
+ std::map<std::pair<int, int>, DecimalV2Value> min_decimalv2_val;
+ std::map<std::pair<int, int>, DecimalV2Value> max_decimalv2_val;
+ std::map<int, int32_t> min_decimal32_val;
+ std::map<int, int32_t> max_decimal32_val;
+ std::map<int, int64_t> min_decimal64_val;
+ std::map<int, int64_t> max_decimal64_val;
+ std::map<int, int128_t> min_decimal128_val;
+ std::map<int, int128_t> max_decimal128_val;
+ std::map<int, wide::Int256> min_decimal256_val;
+ std::map<int, wide::Int256> max_decimal256_val;
+};
+
+// Validate a full (non-const) column without modifying its input tree. Scalar
+// checks reuse input storage; complex types build shells around validated
children.
+Result<ColumnPtr> tablet_validate_column(const ColumnPtr& column, const
DataTypePtr& type,
+ TabletValidateContext& context);
+
+} // namespace doris
diff --git a/be/src/exec/sink/vtablet_block_convertor.cpp
b/be/src/exec/sink/vtablet_block_convertor.cpp
index 134aa2a2140..99e4794a5d4 100644
--- a/be/src/exec/sink/vtablet_block_convertor.cpp
+++ b/be/src/exec/sink/vtablet_block_convertor.cpp
@@ -18,51 +18,30 @@
#include "exec/sink/vtablet_block_convertor.h"
#include <fmt/format.h>
-#include <gen_cpp/FrontendService.h>
#include <glog/logging.h>
-#include <google/protobuf/stubs/common.h>
-#include <algorithm>
#include <memory>
-#include <string>
-#include <unordered_map>
#include <utility>
+#include <vector>
#include "common/compiler_util.h" // IWYU pragma: keep
#include "common/consts.h"
#include "common/status.h"
#include "core/assert_cast.h"
-#include "core/binary_cast.hpp"
#include "core/block/block.h"
#include "core/column/column.h"
-#include "core/column/column_array.h"
#include "core/column/column_const.h"
-#include "core/column/column_decimal.h"
-#include "core/column/column_map.h"
#include "core/column/column_nullable.h"
-#include "core/column/column_string.h"
-#include "core/column/column_struct.h"
-#include "core/data_type/data_type.h"
-#include "core/data_type/data_type_array.h"
-#include "core/data_type/data_type_decimal.h"
-#include "core/data_type/data_type_factory.hpp"
-#include "core/data_type/data_type_map.h"
+#include "core/column/column_vector.h"
#include "core/data_type/data_type_nullable.h"
-#include "core/data_type/data_type_struct.h"
-#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/data_type_number.h"
#include "core/data_type/primitive_type.h"
-#include "core/types.h"
-#include "core/wide_integer_to_string.h"
-#include "exprs/function/function_helpers.h"
-#include "exprs/function/simple_function_factory.h"
-#include "exprs/vexpr.h"
+#include "exec/sink/tablet_column_validator.h"
#include "exprs/vexpr_context.h"
#include "runtime/descriptors.h"
+#include "runtime/runtime_profile.h"
#include "runtime/runtime_state.h"
-#include "service/brpc.h"
-#include "storage/olap_common.h"
-#include "util/brpc_client_cache.h"
-#include "util/thread.h"
+#include "util/defer_op.h"
namespace doris {
@@ -103,7 +82,8 @@ Status
OlapTableBlockConvertor::validate_and_convert_block(RuntimeState* state,
SCOPED_RAW_TIMER(&_validate_data_ns);
_filter_map.clear();
_filter_map.resize(rows, 0);
- auto st = _validate_data(state, block.get(), rows, filtered_rows);
+ _tablet_validate_context.state = state;
+ auto st = _validate_data(block.get(), rows, filtered_rows);
_num_filtered_rows += filtered_rows;
has_filtered_rows = filtered_rows > 0;
if (!st.ok()) {
@@ -136,457 +116,11 @@ void OlapTableBlockConvertor::init_autoinc_info(int64_t
db_id, int64_t table_id,
}
}
-template <bool is_min>
-DecimalV2Value OlapTableBlockConvertor::_get_decimalv2_min_or_max(const
DataTypePtr& type) {
- std::map<std::pair<int, int>, DecimalV2Value>* pmap;
- if constexpr (is_min) {
- pmap = &_min_decimalv2_val;
- } else {
- pmap = &_max_decimalv2_val;
- }
-
- // found
- auto iter = pmap->find(
- {remove_nullable(type)->get_precision(),
remove_nullable(type)->get_scale()});
- if (iter != pmap->end()) {
- return iter->second;
- }
-
- // save min or max DecimalV2Value for next time
- DecimalV2Value value;
- if constexpr (is_min) {
- value.to_min_decimal(type->get_precision(), type->get_scale());
- } else {
- value.to_max_decimal(type->get_precision(), type->get_scale());
- }
- pmap->emplace(std::pair<int, int> {type->get_precision(),
type->get_scale()}, value);
- return value;
-}
-
-template <typename DecimalType, bool IsMin>
-DecimalType OlapTableBlockConvertor::_get_decimalv3_min_or_max(const
DataTypePtr& type) {
- std::map<int, typename DecimalType::NativeType>* pmap;
- if constexpr (std::is_same_v<DecimalType, Decimal32>) {
- pmap = IsMin ? &_min_decimal32_val : &_max_decimal32_val;
- } else if constexpr (std::is_same_v<DecimalType, Decimal64>) {
- pmap = IsMin ? &_min_decimal64_val : &_max_decimal64_val;
- } else if constexpr (std::is_same_v<DecimalType, Decimal128V3>) {
- pmap = IsMin ? &_min_decimal128_val : &_max_decimal128_val;
- } else {
- pmap = IsMin ? &_min_decimal256_val : &_max_decimal256_val;
- }
-
- // found
- auto iter = pmap->find(type->get_precision());
- if (iter != pmap->end()) {
- return DecimalType(iter->second);
- }
-
- DecimalType value;
- if constexpr (IsMin) {
- value = min_decimal_value<DecimalType::PType>(type->get_precision());
- } else {
- value = max_decimal_value<DecimalType::PType>(type->get_precision());
- }
- pmap->emplace(type->get_precision(), value.value);
- return value;
-}
-
-Status OlapTableBlockConvertor::_internal_validate_column(RuntimeState* state,
Block* block,
- const DataTypePtr&
type, ColumnPtr column,
- size_t slot_index,
- fmt::memory_buffer&
error_prefix,
- const size_t
row_count,
-
IColumn::Permutation* rows) {
- DCHECK((rows == nullptr) || (rows->size() == row_count));
- fmt::memory_buffer error_msg;
- auto set_invalid_and_append_error_msg = [&](size_t row) {
- _filter_map[row] = true;
- auto ret = state->append_error_msg_to_file([]() -> std::string {
return ""; },
- [&error_prefix,
&error_msg]() -> std::string {
- return
fmt::to_string(error_prefix) +
-
fmt::to_string(error_msg);
- });
- error_msg.clear();
- return ret;
- };
-
- const auto* column_ptr = check_and_get_column<ColumnNullable>(*column);
- const auto& real_column_ptr =
- column_ptr == nullptr ? column :
(column_ptr->get_nested_column_ptr());
- const auto* null_map = column_ptr == nullptr ? nullptr :
column_ptr->get_null_map_data().data();
- auto need_to_validate = [](size_t j, size_t row, const std::vector<char>&
filter_map,
- const unsigned char* null_map) {
- return !filter_map[row] && (null_map == nullptr || null_map[j] == 0);
- };
-
- // may change orig_column if substring function is performed
- auto string_column_checker = [&state, &error_msg, need_to_validate,
- set_invalid_and_append_error_msg](
- ColumnPtr& orig_column, const
DataTypePtr& orig_type,
- IColumn::Permutation* rows,
- const std::vector<char>& filter_map) {
- int limit = config::string_type_length_soft_limit_bytes;
- int len = -1;
- // when type.len is negative, std::min will return overflow value, so
we need to check it
- const auto* type_str =
-
check_and_get_data_type<DataTypeString>(remove_nullable(orig_type).get());
- if (type_str) {
- if (type_str->len() >= 0) {
- len = type_str->len();
- limit = std::min(limit, type_str->len());
- }
- }
-
- auto tmp_column_ptr =
check_and_get_column<ColumnNullable>(*orig_column);
- auto tmp_real_column_ptr =
- tmp_column_ptr == nullptr ? orig_column :
(tmp_column_ptr->get_nested_column_ptr());
- const auto* column_string = assert_cast<const
ColumnString*>(tmp_real_column_ptr.get());
- const auto* null_map =
- tmp_column_ptr == nullptr ? nullptr :
tmp_column_ptr->get_null_map_data().data();
-
- const auto* __restrict offsets = column_string->get_offsets().data();
- int invalid_count = 0;
- size_t row_count = orig_column->size();
- for (int64_t j = 0; j < row_count; ++j) {
- invalid_count += (offsets[j] - offsets[j - 1]) > limit;
- }
-
- if (invalid_count) {
- // For string column, if in non-strict load mode(for both insert
stmt and stream load),
- // truncate the string to schema len.
- // After truncation, still need to check if byte len of each row
exceed the schema len,
- // because currently the schema len is defined in bytes, and
substring works by unit of chars.
- // This is a workaround for now, need to improve it after better
support of multi-byte chars.
- if (type_str && !state->enable_insert_strict()) {
- ColumnsWithTypeAndName argument_template;
- auto input_type = remove_nullable(orig_type);
- auto pos_type = DataTypeFactory::instance().create_data_type(
- FieldType::OLAP_FIELD_TYPE_INT, 0, 0);
- auto len_type = DataTypeFactory::instance().create_data_type(
- FieldType::OLAP_FIELD_TYPE_INT, 0, 0);
- argument_template.emplace_back(nullptr, input_type, "string
column");
- argument_template.emplace_back(nullptr, pos_type, "pos
column");
- argument_template.emplace_back(nullptr, len_type, "len
column");
- auto func = SimpleFunctionFactory::instance().get_function(
- "substring", argument_template, input_type, {},
state->be_exec_version());
- if (!func) {
- return Status::InternalError("get function substring
failed");
- }
- auto pos_column = pos_type->create_column_const(row_count,
to_field<TYPE_INT>(1));
- auto len_column =
- len_type->create_column_const(row_count,
to_field<TYPE_INT>(limit));
- Block tmp_block({{remove_nullable(orig_column), input_type,
"string column"},
- {pos_column, pos_type, "pos"},
- {len_column, len_type, "len"},
- {nullptr, input_type, "result"}});
- RETURN_IF_ERROR(func->execute(nullptr, tmp_block, {0, 1, 2},
3, row_count));
- auto result_column =
-
IColumn::mutate(std::move(tmp_block.get_by_position(3).column));
- if (orig_column->is_nullable()) {
- orig_column = ColumnNullable::create(
- std::move(result_column),
-
IColumn::mutate(tmp_column_ptr->get_null_map_column_ptr()));
- } else {
- orig_column = std::move(result_column);
- }
- tmp_column_ptr =
check_and_get_column<ColumnNullable>(*orig_column);
- tmp_real_column_ptr = tmp_column_ptr == nullptr
- ? orig_column
- :
tmp_column_ptr->get_nested_column_ptr();
- column_string = assert_cast<const
ColumnString*>(tmp_real_column_ptr.get());
- null_map = tmp_column_ptr == nullptr ? nullptr
- :
tmp_column_ptr->get_null_map_data().data();
- }
- for (size_t j = 0; j < row_count; ++j) {
- auto row = rows ? (*rows)[j] : j;
- if (need_to_validate(j, row, filter_map, null_map)) {
- auto str_val = column_string->get_data_at(j);
- bool invalid = str_val.size > limit;
- if (invalid) {
- if (str_val.size > len) {
- fmt::format_to(error_msg, "{}",
- "the length of input is too long
than schema. ");
- fmt::format_to(error_msg, "first 32 bytes of input
str: [{}] ",
- str_val.to_prefix(32));
- fmt::format_to(error_msg, "schema length: {}; ",
len);
- fmt::format_to(error_msg, "actual length: {}; ",
str_val.size);
- } else if (str_val.size > limit) {
- fmt::format_to(
- error_msg, "{}",
- "the length of input string is too long
than vec schema. ");
- fmt::format_to(error_msg, "first 32 bytes of input
str: [{}] ",
- str_val.to_prefix(32));
- fmt::format_to(error_msg, "schema length: {}; ",
len);
- fmt::format_to(error_msg, "limit length: {}; ",
limit);
- fmt::format_to(error_msg, "actual length: {}; ",
str_val.size);
- }
- RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
- }
- }
- }
- }
- return Status::OK();
- };
-
- switch (type->get_primitive_type()) {
- case TYPE_CHAR:
- case TYPE_VARCHAR:
- case TYPE_STRING: {
- RETURN_IF_ERROR(string_column_checker(column, type, rows,
_filter_map));
- block->get_by_position(slot_index).column = std::move(column);
- break;
- }
- case TYPE_JSONB: {
- const auto* column_string = assert_cast<const
ColumnString*>(real_column_ptr.get());
- for (size_t j = 0; j < row_count; ++j) {
- if (!_filter_map[j]) {
- if (type->is_nullable() && column_ptr &&
column_ptr->is_null_at(j)) {
- continue;
- }
- auto str_val = column_string->get_data_at(j);
- bool invalid = str_val.size == 0;
- if (invalid) {
- error_msg.clear();
- fmt::format_to(error_msg, "{}", "jsonb with size 0 is
invalid");
- RETURN_IF_ERROR(set_invalid_and_append_error_msg(j));
- }
- }
- }
- break;
- }
- case TYPE_DECIMALV2: {
- // column_decimal utilizes the ColumnPtr from the block* block in
_validate_data and can be modified.
- auto* column_decimal = const_cast<ColumnDecimal128V2*>(
- assert_cast<const ColumnDecimal128V2*>(real_column_ptr.get()));
- const auto& max_decimalv2 = _get_decimalv2_min_or_max<false>(type);
- const auto& min_decimalv2 = _get_decimalv2_min_or_max<true>(type);
- for (size_t j = 0; j < row_count; ++j) {
- auto row = rows ? (*rows)[j] : j;
- if (need_to_validate(j, row, _filter_map, null_map)) {
- auto dec_val = binary_cast<Int128,
DecimalV2Value>(column_decimal->get_data()[j]);
- bool invalid = false;
-
- if (dec_val.greater_than_scale(type->get_scale())) {
- auto code =
- dec_val.round(&dec_val,
remove_nullable(type)->get_scale(), HALF_UP);
- column_decimal->get_data()[j] = dec_val;
-
- if (code != E_DEC_OK) {
- fmt::format_to(error_msg, "round one decimal
failed.value={}; ",
- dec_val.to_string());
- invalid = true;
- }
- }
- if (dec_val > max_decimalv2 || dec_val < min_decimalv2) {
- fmt::format_to(error_msg, "{}", "decimal value is not
valid for definition");
- fmt::format_to(error_msg, ", value={}",
dec_val.to_string());
- fmt::format_to(error_msg, ", precision={}, scale={}",
type->get_precision(),
- type->get_scale());
- fmt::format_to(error_msg, ", min={}, max={}; ",
min_decimalv2.to_string(),
- max_decimalv2.to_string());
- invalid = true;
- }
-
- if (invalid) {
- RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
- }
- }
- }
- break;
- }
- case TYPE_DECIMAL32: {
-#define CHECK_VALIDATION_FOR_DECIMALV3(DecimalType)
\
- auto column_decimal =
\
- assert_cast<const
ColumnDecimal<DecimalType::PType>*>(real_column_ptr.get()); \
- const auto& max_decimal = _get_decimalv3_min_or_max<DecimalType,
false>(type); \
- const auto& min_decimal = _get_decimalv3_min_or_max<DecimalType,
true>(type); \
- const auto* __restrict datas = column_decimal->get_data().data();
\
- int invalid_count = 0;
\
- for (int j = 0; j < row_count; ++j) {
\
- const auto dec_val = datas[j];
\
- invalid_count += dec_val > max_decimal || dec_val < min_decimal;
\
- }
\
- if (invalid_count) {
\
- for (size_t j = 0; j < row_count; ++j) {
\
- auto row = rows ? (*rows)[j] : j;
\
- if (need_to_validate(j, row, _filter_map, null_map)) {
\
- auto dec_val = column_decimal->get_data()[j];
\
- bool invalid = false;
\
- if (dec_val > max_decimal || dec_val < min_decimal) {
\
- fmt::format_to(error_msg, "{}", "decimal value is not
valid for definition"); \
- fmt::format_to(error_msg, ", value={}", dec_val.value);
\
- fmt::format_to(error_msg, ", precision={}, scale={}",
type->get_precision(), \
- type->get_scale());
\
- fmt::format_to(error_msg, ", min={}, max={}; ",
min_decimal.value, \
- max_decimal.value);
\
- invalid = true;
\
- }
\
- if (invalid) {
\
- RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
\
- }
\
- }
\
- }
\
- }
- CHECK_VALIDATION_FOR_DECIMALV3(Decimal32);
- break;
- }
- case TYPE_DECIMAL64: {
- CHECK_VALIDATION_FOR_DECIMALV3(Decimal64);
- break;
- }
- case TYPE_DECIMAL128I: {
- CHECK_VALIDATION_FOR_DECIMALV3(Decimal128V3);
- break;
- }
- case TYPE_DECIMAL256: {
- CHECK_VALIDATION_FOR_DECIMALV3(Decimal256);
- break;
- }
-#undef CHECK_VALIDATION_FOR_DECIMALV3
- case TYPE_ARRAY: {
- const auto* column_array = assert_cast<const
ColumnArray*>(real_column_ptr.get());
- const auto* type_array = assert_cast<const
DataTypeArray*>(remove_nullable(type).get());
- auto nested_type = type_array->get_nested_type();
- const auto& offsets = column_array->get_offsets();
- IColumn::Permutation permutation(offsets.back());
- for (size_t r = 0; r < row_count; ++r) {
- for (size_t c = offsets[r - 1]; c < offsets[r]; ++c) {
- permutation[c] = rows ? (*rows)[r] : r;
- }
- }
- fmt::format_to(error_prefix, "ARRAY type failed: ");
- auto data_column_ptr = column_array->get_data_ptr();
- switch (nested_type->get_primitive_type()) {
- case TYPE_CHAR:
- case TYPE_VARCHAR:
- case TYPE_STRING: {
- RETURN_IF_ERROR(
- string_column_checker(data_column_ptr, nested_type,
&permutation, _filter_map));
- const_cast<ColumnArray*>(column_array)->get_data_ptr() =
std::move(data_column_ptr);
- break;
- }
- default:
- RETURN_IF_ERROR(_validate_column(state, block, nested_type,
data_column_ptr, slot_index,
- error_prefix, permutation.size(),
&permutation));
- break;
- }
- break;
- }
- case TYPE_MAP: {
- const auto* column_map = assert_cast<const
ColumnMap*>(real_column_ptr.get());
- // column_map utilizes the ColumnPtr from the block* block in
_validate_data and can be modified.
-
RETURN_IF_ERROR((const_cast<ColumnMap*>(column_map))->deduplicate_keys(true));
-
- const auto* type_map = assert_cast<const
DataTypeMap*>(remove_nullable(type).get());
- auto key_type = type_map->get_key_type();
- auto val_type = type_map->get_value_type();
- const auto& offsets = column_map->get_offsets();
- IColumn::Permutation permutation(offsets.back());
- for (size_t r = 0; r < row_count; ++r) {
- for (size_t c = offsets[r - 1]; c < offsets[r]; ++c) {
- permutation[c] = rows ? (*rows)[r] : r;
- }
- }
-
- fmt::format_to(error_prefix, "MAP type failed: ");
- switch (key_type->get_primitive_type()) {
- case TYPE_CHAR:
- case TYPE_VARCHAR:
- case TYPE_STRING: {
- auto key_column_ptr = column_map->get_keys_ptr();
- RETURN_IF_ERROR(
- string_column_checker(key_column_ptr, key_type,
&permutation, _filter_map));
- const_cast<ColumnMap*>(column_map)->get_keys_ptr() =
std::move(key_column_ptr);
- break;
- }
- default:
- RETURN_IF_ERROR(_validate_column(state, block, key_type,
column_map->get_keys_ptr(),
- slot_index, error_prefix,
permutation.size(),
- &permutation));
- break;
- }
-
- switch (val_type->get_primitive_type()) {
- case TYPE_CHAR:
- case TYPE_VARCHAR:
- case TYPE_STRING: {
- auto value_column_ptr = column_map->get_values_ptr();
- RETURN_IF_ERROR(
- string_column_checker(value_column_ptr, val_type,
&permutation, _filter_map));
- const_cast<ColumnMap*>(column_map)->get_values_ptr() =
std::move(value_column_ptr);
- break;
- }
- default:
- RETURN_IF_ERROR(_validate_column(state, block, val_type,
column_map->get_values_ptr(),
- slot_index, error_prefix,
permutation.size(),
- &permutation));
- break;
- }
- break;
- }
- case TYPE_STRUCT: {
- const auto column_struct = assert_cast<const
ColumnStruct*>(real_column_ptr.get());
- const auto* type_struct = assert_cast<const
DataTypeStruct*>(remove_nullable(type).get());
- DCHECK(type_struct->get_elements().size() ==
column_struct->tuple_size());
- fmt::format_to(error_prefix, "STRUCT type failed: ");
- for (size_t sc = 0; sc < column_struct->tuple_size(); ++sc) {
- auto element_type = type_struct->get_element(sc);
- switch (element_type->get_primitive_type()) {
- case TYPE_CHAR:
- case TYPE_VARCHAR:
- case TYPE_STRING: {
- auto element_column_ptr = column_struct->get_column_ptr(sc);
- RETURN_IF_ERROR(string_column_checker(element_column_ptr,
element_type, nullptr,
- _filter_map));
- const_cast<ColumnStruct*>(column_struct)->get_column_ptr(sc) =
- std::move(element_column_ptr);
- break;
- }
- default:
- RETURN_IF_ERROR(_validate_column(state, block,
type_struct->get_element(sc),
-
column_struct->get_column_ptr(sc), slot_index,
- error_prefix,
-
column_struct->get_column_ptr(sc)->size()));
- break;
- }
- }
- break;
- }
- case TYPE_AGG_STATE: {
- auto* column_string =
check_and_get_column<ColumnString>(*real_column_ptr);
- if (column_string) {
- RETURN_IF_ERROR(string_column_checker(column, type, rows,
_filter_map));
- }
- break;
- }
- default:
- break;
- }
-
- // Dispose the column should do not contain the NULL value
- // Only two case:
- // 1. column is nullable but the desc is not nullable
- // 2. desc->type is BITMAP
- if ((!type->is_nullable() || type->get_primitive_type() == TYPE_BITMAP) &&
column_ptr) {
- for (int j = 0; j < row_count; ++j) {
- auto row = rows ? (*rows)[j] : j;
- if (null_map[j] && !_filter_map[row]) {
- fmt::format_to(error_msg, "null value for not null column,
type={}",
- type->get_name());
- RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
- }
- }
- }
-
- return Status::OK();
-}
-
-Status OlapTableBlockConvertor::_validate_data(RuntimeState* state, Block*
block, const size_t rows,
+Status OlapTableBlockConvertor::_validate_data(Block* block, const size_t rows,
int& filtered_rows) {
filtered_rows = 0;
Defer defer {[&] {
- for (int i = 0; i < rows; ++i) {
+ for (size_t i = 0; i < rows; ++i) {
filtered_rows += _filter_map[i];
}
}};
@@ -595,12 +129,13 @@ Status
OlapTableBlockConvertor::_validate_data(RuntimeState* state, Block* block
// VRowDistribution's _deal_missing_map() relies on these slots being
full columns.
block->get_by_position(i).column =
block->get_by_position(i).column->convert_to_full_column_if_const();
- const auto& column = block->get_by_position(i).column;
+ auto& column = block->get_by_position(i).column;
- fmt::memory_buffer error_prefix;
- fmt::format_to(error_prefix, "column_name[{}], ", desc->col_name());
- RETURN_IF_ERROR(
- _validate_column(state, block, desc->type(), column, i,
error_prefix, rows));
+ DCHECK_EQ(column->size(), rows);
+ _tablet_validate_context.error_prefix.clear();
+ fmt::format_to(_tablet_validate_context.error_prefix,
"column_name[{}], ",
+ desc->col_name());
+ column = DORIS_TRY(tablet_validate_column(column, desc->type(),
_tablet_validate_context));
}
return Status::OK();
}
diff --git a/be/src/exec/sink/vtablet_block_convertor.h
b/be/src/exec/sink/vtablet_block_convertor.h
index f529b7da869..afd2f1136a7 100644
--- a/be/src/exec/sink/vtablet_block_convertor.h
+++ b/be/src/exec/sink/vtablet_block_convertor.h
@@ -22,14 +22,13 @@
// IWYU pragma: no_include <bits/chrono.h>
#include <chrono> // IWYU pragma: keep
-#include <map>
#include "common/status.h"
#include "core/block/block.h"
#include "core/column/column.h"
#include "core/data_type/data_type_factory.hpp"
-#include "core/value/decimalv2_value.h"
#include "exec/sink/autoinc_buffer.h"
+#include "exec/sink/tablet_column_validator.h"
#include "exprs/vexpr_fwd.h"
#include "runtime/descriptors.h"
#include "util/bitmap.h"
@@ -39,7 +38,7 @@ namespace doris {
class OlapTableBlockConvertor {
public:
OlapTableBlockConvertor(TupleDescriptor* output_tuple_desc)
- : _output_tuple_desc(output_tuple_desc) {}
+ : _output_tuple_desc(output_tuple_desc),
_tablet_validate_context(_filter_map) {}
Status validate_and_convert_block(RuntimeState* state, Block* input_block,
std::shared_ptr<Block>& block,
@@ -59,30 +58,10 @@ public:
AutoIncIDAllocator& auto_inc_id_allocator() { return
_auto_inc_id_allocator; }
private:
- template <bool is_min>
- DecimalV2Value _get_decimalv2_min_or_max(const DataTypePtr& type);
-
- template <typename DecimalType, bool IsMin>
- DecimalType _get_decimalv3_min_or_max(const DataTypePtr& type);
-
- Status _validate_column(RuntimeState* state, Block* block, const
DataTypePtr& type,
- ColumnPtr column, size_t slot_index,
fmt::memory_buffer& error_prefix,
- const size_t row_count, IColumn::Permutation* rows
= nullptr) {
- RETURN_IF_CATCH_EXCEPTION({
- return _internal_validate_column(state, block, type, column,
slot_index, error_prefix,
- row_count, rows);
- });
- }
-
- Status _internal_validate_column(RuntimeState* state, Block* block, const
DataTypePtr& type,
- ColumnPtr column, size_t slot_index,
- fmt::memory_buffer& error_prefix, const
size_t row_count,
- IColumn::Permutation* rows = nullptr);
-
// make input data valid for OLAP table
// return number of invalid/filtered rows.
// invalid row number is set in Bitmap
- Status _validate_data(RuntimeState* state, Block* block, const size_t
rows, int& filtered_rows);
+ Status _validate_data(Block* block, const size_t rows, int& filtered_rows);
// some output column of output expr may have different nullable property
with dest slot desc
// so here need to do the convert operation
@@ -94,20 +73,10 @@ private:
TupleDescriptor* _output_tuple_desc = nullptr;
- std::map<std::pair<int, int>, DecimalV2Value> _max_decimalv2_val;
- std::map<std::pair<int, int>, DecimalV2Value> _min_decimalv2_val;
-
- std::map<int, int32_t> _max_decimal32_val;
- std::map<int, int32_t> _min_decimal32_val;
- std::map<int, int64_t> _max_decimal64_val;
- std::map<int, int64_t> _min_decimal64_val;
- std::map<int, int128_t> _max_decimal128_val;
- std::map<int, int128_t> _min_decimal128_val;
- std::map<int, wide::Int256> _max_decimal256_val;
- std::map<int, wide::Int256> _min_decimal256_val;
-
std::vector<char> _filter_map;
+ TabletValidateContext _tablet_validate_context;
+
int64_t _validate_data_ns = 0;
int64_t _num_filtered_rows = 0;
diff --git a/be/test/exec/sink/tablet_column_validator_test.cpp
b/be/test/exec/sink/tablet_column_validator_test.cpp
new file mode 100644
index 00000000000..b6b954a8a5b
--- /dev/null
+++ b/be/test/exec/sink/tablet_column_validator_test.cpp
@@ -0,0 +1,511 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/sink/tablet_column_validator.h"
+
+#include <gtest/gtest.h>
+
+#include <initializer_list>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "agent/be_exec_version_manager.h"
+#include "core/assert_cast.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_jsonb.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "runtime/runtime_state.h"
+#include "storage/olap_common.h"
+
+namespace doris {
+namespace {
+
+// Capture leaf storage without keeping owning pointers. Complex shells may
change.
+struct StorageSnapshot {
+ const IColumn* column;
+ const void* data;
+};
+
+std::vector<StorageSnapshot> snapshot(const IColumn& column) {
+ std::vector<StorageSnapshot> result;
+ const auto visit = [&](const auto& self, const IColumn& current) -> void {
+ const void* data = nullptr;
+ if (const auto* ints = check_and_get_column<ColumnInt32>(current)) {
+ data = ints->get_data().data();
+ } else if (const auto* bytes =
check_and_get_column<ColumnUInt8>(current)) {
+ data = bytes->get_data().data();
+ } else if (const auto* offsets =
+
check_and_get_column<ColumnArray::ColumnOffsets>(current)) {
+ data = offsets->get_data().data();
+ } else if (const auto* strings =
check_and_get_column<ColumnString>(current)) {
+ data = strings->get_chars().data();
+ } else if (const auto* decimals =
check_and_get_column<ColumnDecimal128V2>(current)) {
+ data = decimals->get_data().data();
+ }
+ result.push_back({¤t, data});
+ if (const auto* strings = check_and_get_column<ColumnString>(current))
{
+ result.push_back({¤t, strings->get_offsets().data()});
+ } else if (const auto* nullable =
check_and_get_column<ColumnNullable>(current)) {
+ self(self, nullable->get_nested_column());
+ self(self, nullable->get_null_map_column());
+ } else if (const auto* array =
check_and_get_column<ColumnArray>(current)) {
+ self(self, array->get_data());
+ self(self, array->get_offsets_column());
+ } else if (const auto* map = check_and_get_column<ColumnMap>(current))
{
+ self(self, map->get_keys());
+ self(self, map->get_values());
+ self(self, *map->get_offsets_ptr());
+ } else if (const auto* tuple =
check_and_get_column<ColumnStruct>(current)) {
+ for (size_t i = 0; i < tuple->tuple_size(); ++i) {
+ self(self, tuple->get_column(i));
+ }
+ }
+ };
+ visit(visit, column);
+ return result;
+}
+
+void expect_same_storage(const std::vector<StorageSnapshot>& before, const
IColumn& column) {
+ const auto after = snapshot(column);
+ ASSERT_EQ(before.size(), after.size());
+ for (size_t i = 0; i < before.size(); ++i) {
+ if (before[i].data) {
+ EXPECT_EQ(before[i].column, after[i].column) << "leaf " << i;
+ EXPECT_EQ(before[i].data, after[i].data) << "leaf " << i;
+ }
+ }
+}
+
+ColumnInt32::MutablePtr ints(std::initializer_list<Int32> values) {
+ auto column = ColumnInt32::create();
+ for (auto value : values) {
+ column->insert_value(value);
+ }
+ return column;
+}
+
+ColumnArray::ColumnOffsets::MutablePtr offsets(std::initializer_list<UInt64>
values) {
+ auto column = ColumnArray::ColumnOffsets::create();
+ for (auto value : values) {
+ column->insert_value(value);
+ }
+ return column;
+}
+
+ColumnString::MutablePtr strings(std::initializer_list<std::string> values) {
+ auto column = ColumnString::create();
+ for (const auto& value : values) {
+ column->insert_data(value.data(), value.size());
+ }
+ return column;
+}
+
+DataTypePtr decimal_v2_type() {
+ return
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_DECIMAL,
9, 2);
+}
+
+TQueryOptions options(bool strict) {
+ TQueryOptions options;
+ options.__set_enable_insert_strict(strict);
+ options.__set_be_exec_version(BeExecVersionManager::get_newest_version());
+ return options;
+}
+
+} // namespace
+
+class TabletColumnValidatorTest : public testing::Test {
+protected:
+ TabletColumnValidatorTest() : state(options(true), TQueryGlobals {}),
context(filter_map) {
+ context.state = &state;
+ }
+
+ Result<ColumnPtr> validate(const ColumnPtr& column, const DataTypePtr&
type) {
+ filter_map.assign(column->size(), 0);
+ return tablet_validate_column(column, type, context);
+ }
+
+ RuntimeState state;
+ std::vector<char> filter_map;
+ TabletValidateContext context;
+};
+
+TEST_F(TabletColumnValidatorTest,
UniqueScalarKeepsObjectBufferAndReferenceCount) {
+ ColumnPtr column = ints({7, 8});
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, std::make_shared<DataTypeInt32>()));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(column->use_count(), 1);
+}
+
+TEST_F(TabletColumnValidatorTest, NullableArrayStructReusesLeafStorage) {
+ ColumnPtr field = ColumnNullable::create(ints({7, 8}),
ColumnUInt8::create(2, 0));
+ auto tuple = ColumnStruct::create(Columns {std::move(field), strings({"a",
"b"})});
+ auto array = ColumnArray::create(std::move(tuple), offsets({0, 2}));
+ ColumnPtr column = ColumnNullable::create(std::move(array),
ColumnUInt8::create(2, 0));
+ auto tuple_type = std::make_shared<DataTypeStruct>(DataTypes {
+ make_nullable(std::make_shared<DataTypeInt32>()),
std::make_shared<DataTypeString>()});
+ auto type = make_nullable(std::make_shared<DataTypeArray>(tuple_type));
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 0}));
+}
+
+TEST_F(TabletColumnValidatorTest, MapWithoutDuplicatesReusesLeafStorage) {
+ ColumnPtr column = ColumnMap::create(ints({1, 2}), ints({10, 20}),
offsets({2}));
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeInt32>());
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+}
+
+TEST_F(TabletColumnValidatorTest, StructReusesSharedReadOnlyFields) {
+ ColumnPtr shared = ints({7});
+ ColumnPtr column = ColumnStruct::create(Columns {shared, ints({8}),
strings({"a"})});
+ const auto& tuple_before = assert_cast<const ColumnStruct&>(*column);
+ auto second_before = snapshot(tuple_before.get_column(1));
+ auto third_before = snapshot(tuple_before.get_column(2));
+ const auto* parent_before = column.get();
+ const auto* shared_data = assert_cast<const
ColumnInt32&>(*shared).get_data().data();
+ auto type = std::make_shared<DataTypeStruct>(DataTypes
{std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeString>()});
+
+ column = TEST_TRY(validate(column, type));
+ const auto& tuple = assert_cast<const ColumnStruct&>(*column);
+ EXPECT_NE(column.get(), parent_before);
+ EXPECT_EQ(&tuple.get_column(0), shared.get());
+ EXPECT_EQ(assert_cast<const
ColumnInt32&>(tuple.get_column(0)).get_data().data(), shared_data);
+ EXPECT_EQ(assert_cast<const
ColumnInt32&>(tuple.get_column(0)).get_data()[0], 7);
+ EXPECT_EQ(shared->use_count(), 2);
+ expect_same_storage(second_before, tuple.get_column(1));
+ expect_same_storage(third_before, tuple.get_column(2));
+}
+
+TEST_F(TabletColumnValidatorTest,
DecimalV2PreservesLegacyScaleAndUniqueStorage) {
+ auto decimals = ColumnDecimal128V2::create(0, 9);
+
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+ ColumnPtr column = std::move(decimals);
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, decimal_v2_type()));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(assert_cast<const
ColumnDecimal128V2&>(*column).get_data()[0].value(),
+ DecimalV2Value(std::string("1.235")).value());
+}
+
+TEST_F(TabletColumnValidatorTest,
DecimalV2PrecisionEqualsScalePreservesLegacyValue) {
+ auto decimals = ColumnDecimal128V2::create(0, 9);
+
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+ ColumnPtr column = std::move(decimals);
+ auto before = snapshot(*column);
+ // DECIMAL(2,2) still uses DecimalV2 execution precision 27 and scale 9.
+ auto type =
+
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_DECIMAL,
2, 2);
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(assert_cast<const
ColumnDecimal128V2&>(*column).get_data()[0].value(),
+ DecimalV2Value(std::string("1.235")).value());
+}
+
+TEST_F(TabletColumnValidatorTest, SharedNullableDecimalV2PreservesLegacyValue)
{
+ auto decimals = ColumnDecimal128V2::create(0, 9);
+
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+ ColumnPtr column = ColumnNullable::create(std::move(decimals),
ColumnUInt8::create(1, 0));
+ ColumnPtr original = column;
+ auto before = snapshot(*original);
+ column = TEST_TRY(validate(column, make_nullable(decimal_v2_type())));
+ const auto& result = assert_cast<const ColumnNullable&>(*column);
+ EXPECT_EQ(assert_cast<const
ColumnDecimal128V2&>(result.get_nested_column())
+ .get_data()[0]
+ .value(),
+ DecimalV2Value(std::string("1.235")).value());
+ const auto& old = assert_cast<const ColumnNullable&>(*original);
+ EXPECT_EQ(assert_cast<const
ColumnDecimal128V2&>(old.get_nested_column()).get_data()[0].value(),
+ DecimalV2Value(std::string("1.235")).value());
+ expect_same_storage(before, *original);
+}
+
+TEST_F(TabletColumnValidatorTest, SharedMapColumnIsIsolated) {
+ ColumnPtr column = ColumnMap::create(ints({7, 7}), ints({10, 20}),
offsets({2}));
+ ColumnPtr original = column;
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeInt32>());
+ column = TEST_TRY(validate(column, type));
+ const auto& result = assert_cast<const ColumnMap&>(*column);
+ const auto& old = assert_cast<const ColumnMap&>(*original);
+ EXPECT_EQ(result.get_keys().size(), 1);
+ EXPECT_EQ(assert_cast<const
ColumnInt32&>(result.get_values()).get_data()[0], 20);
+ EXPECT_EQ(old.get_keys().size(), 2);
+ EXPECT_EQ(old.get_values().size(), 2);
+ EXPECT_EQ(old.get_offsets()[0], 2);
+}
+
+TEST_F(TabletColumnValidatorTest, StructInsideArrayReportsOriginalRow) {
+ auto tuple = ColumnStruct::create(Columns {strings({""})});
+ ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({0, 1}));
+ auto tuple_type =
+ std::make_shared<DataTypeStruct>(DataTypes
{std::make_shared<DataTypeJsonb>()});
+ column = TEST_TRY(validate(column,
std::make_shared<DataTypeArray>(tuple_type)));
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+ EXPECT_EQ(context.error_prefix.size(), 0);
+}
+
+TEST_F(TabletColumnValidatorTest, NullArraySkipsInvalidElementPayload) {
+ auto array = ColumnArray::create(strings({"", ""}), offsets({1, 2}));
+ auto nulls = ColumnUInt8::create();
+ nulls->insert_value(1);
+ nulls->insert_value(0);
+ ColumnPtr column = ColumnNullable::create(std::move(array),
std::move(nulls));
+ auto type =
make_nullable(std::make_shared<DataTypeArray>(std::make_shared<DataTypeJsonb>()));
+ column = TEST_TRY(validate(column, type));
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+}
+
+TEST_F(TabletColumnValidatorTest, ParentAndChildNullMapsAreCombined) {
+ ColumnPtr child = ColumnNullable::create(strings({"", "", ""}),
ColumnUInt8::create(3, 0));
+ auto& child_nulls =
+
assert_cast<ColumnNullable&>(child->assert_mutable_ref()).get_null_map_data();
+ child_nulls[1] = 1;
+ auto tuple = ColumnStruct::create(Columns {std::move(child)});
+ auto parent_nulls = ColumnUInt8::create();
+ parent_nulls->insert_value(1);
+ parent_nulls->insert_value(0);
+ parent_nulls->insert_value(0);
+ ColumnPtr column = ColumnNullable::create(std::move(tuple),
std::move(parent_nulls));
+ auto type = make_nullable(std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(std::make_shared<DataTypeJsonb>())}));
+ column = TEST_TRY(validate(column, type));
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 0, 1}));
+}
+
+TEST_F(TabletColumnValidatorTest, NullParentSkipsNotNullChildViolation) {
+ ColumnPtr child = ColumnNullable::create(ints({0}), ColumnUInt8::create(1,
1));
+ auto tuple = ColumnStruct::create(Columns {std::move(child)});
+ ColumnPtr column = ColumnNullable::create(std::move(tuple),
ColumnUInt8::create(1, 1));
+ auto type = make_nullable(
+ std::make_shared<DataTypeStruct>(DataTypes
{std::make_shared<DataTypeInt32>()}));
+ column = TEST_TRY(validate(column, type));
+ EXPECT_EQ(filter_map, (std::vector<char> {0}));
+}
+
+TEST_F(TabletColumnValidatorTest, NonStrictTruncationReplacesOnlyNestedString)
{
+ RuntimeState non_strict(options(false), TQueryGlobals {});
+ context.state = &non_strict;
+ ColumnPtr column = ColumnNullable::create(strings({"abcd"}),
ColumnUInt8::create(1, 0));
+ const auto* original_column = column.get();
+ const auto& old = assert_cast<const ColumnNullable&>(*column);
+ const auto* original_nulls = old.get_null_map_data().data();
+ auto type = make_nullable(std::make_shared<DataTypeString>(2,
TYPE_VARCHAR));
+ column = TEST_TRY(validate(column, type));
+ const auto& result = assert_cast<const ColumnNullable&>(*column);
+ EXPECT_NE(column.get(), original_column);
+ EXPECT_EQ(result.get_null_map_data().data(), original_nulls);
+ EXPECT_EQ(result.get_nested_column().get_data_at(0).to_string(), "ab");
+ EXPECT_EQ(filter_map, (std::vector<char> {0}));
+}
+
+TEST_F(TabletColumnValidatorTest, StrictStringLengthFailurePreservesStorage) {
+ ColumnPtr column = strings({"abcd"});
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, std::make_shared<DataTypeString>(2,
TYPE_VARCHAR)));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest, EmptyNestedColumnsAreValid) {
+ ColumnPtr column = ColumnArray::create(strings({}), offsets({0, 0}));
+ auto before = snapshot(*column);
+ auto type =
std::make_shared<DataTypeArray>(std::make_shared<DataTypeJsonb>());
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 0}));
+}
+
+TEST_F(TabletColumnValidatorTest, NestedNullableMapReusesLeafStorage) {
+ auto inner = ColumnMap::create(ints({3, 4}), ints({30, 40}), offsets({2}));
+ auto nullable = ColumnNullable::create(std::move(inner),
ColumnUInt8::create(1, 0));
+ ColumnPtr column = ColumnMap::create(ints({1}), std::move(nullable),
offsets({1}));
+ auto inner_type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeInt32>());
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+ make_nullable(inner_type));
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+}
+
+TEST_F(TabletColumnValidatorTest,
SharedNestedNullableMapDeduplicatesWithoutChangingSource) {
+ auto inner = ColumnMap::create(ints({3, 3}), ints({30, 40}), offsets({2}));
+ auto nullable = ColumnNullable::create(std::move(inner),
ColumnUInt8::create(1, 0));
+ ColumnPtr column = ColumnMap::create(ints({1}), std::move(nullable),
offsets({1}));
+ ColumnPtr original = column;
+ auto inner_type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeInt32>());
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+ make_nullable(inner_type));
+ column = TEST_TRY(validate(column, type));
+ const auto& result = assert_cast<const ColumnMap&>(*column);
+ const auto& result_nullable = assert_cast<const
ColumnNullable&>(result.get_values());
+ const auto& result_inner = assert_cast<const
ColumnMap&>(result_nullable.get_nested_column());
+ EXPECT_EQ(result_inner.get_keys().size(), 1);
+ EXPECT_EQ(assert_cast<const
ColumnInt32&>(result_inner.get_values()).get_data()[0], 40);
+ const auto& old = assert_cast<const ColumnMap&>(*original);
+ const auto& old_nullable = assert_cast<const
ColumnNullable&>(old.get_values());
+ EXPECT_EQ(assert_cast<const
ColumnMap&>(old_nullable.get_nested_column()).get_keys().size(), 2);
+}
+
+TEST_F(TabletColumnValidatorTest, MapDeduplicationRebuildsElementRowMapping) {
+ ColumnPtr column =
+ ColumnMap::create(ints({1, 1, 2}), strings({"", "valid", ""}),
offsets({2, 3}));
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+
std::make_shared<DataTypeJsonb>());
+ column = TEST_TRY(validate(column, type));
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+ const auto& result = assert_cast<const ColumnMap&>(*column);
+ EXPECT_EQ(result.get_offsets()[0], 1);
+ EXPECT_EQ(result.get_offsets()[1], 2);
+}
+
+TEST_F(TabletColumnValidatorTest, NotNullRejectsNullRow) {
+ ColumnPtr column = ColumnNullable::create(ints({0}),
ColumnUInt8::create(1, 1));
+ column = TEST_TRY(validate(column, std::make_shared<DataTypeInt32>()));
+ EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest, DecimalV3RangeChecksAllWidths) {
+ const auto check = [this]<PrimitiveType T>() {
+ using ColumnType = ColumnDecimal<T>;
+ using Decimal = typename PrimitiveTypeTraits<T>::CppType;
+ auto values = ColumnType::create(0, 0);
+ values->get_data().push_back(Decimal(9));
+ values->get_data().push_back(Decimal(10));
+ values->get_data().push_back(Decimal(-10));
+ ColumnPtr column = std::move(values);
+ const auto* object = column.get();
+ const auto* data = assert_cast<const
ColumnType&>(*column).get_data().data();
+ auto result = validate(column, std::make_shared<DataTypeDecimal<T>>(1,
0));
+ ASSERT_TRUE(result.has_value()) << result.error();
+ EXPECT_EQ(result.value().get(), object);
+ EXPECT_EQ(assert_cast<const
ColumnType&>(*result.value()).get_data().data(), data);
+ EXPECT_EQ(filter_map, (std::vector<char> {0, 1, 1}));
+ };
+ check.template operator()<TYPE_DECIMAL32>();
+ check.template operator()<TYPE_DECIMAL64>();
+ check.template operator()<TYPE_DECIMAL128I>();
+ check.template operator()<TYPE_DECIMAL256>();
+}
+
+TEST_F(TabletColumnValidatorTest, NonNullableInputAcceptsNullableTargetType) {
+ auto tuple = ColumnStruct::create(Columns {strings({"abc"})});
+ ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({1}));
+ auto tuple_type = make_nullable(std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(std::make_shared<DataTypeString>(2,
TYPE_VARCHAR))}));
+ auto type = make_nullable(std::make_shared<DataTypeArray>(tuple_type));
+ auto before = snapshot(*column);
+ column = TEST_TRY(validate(column, type));
+ expect_same_storage(before, *column);
+ EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest,
NestedReplacementBuildsShellsAndPreservesSource) {
+ RuntimeState non_strict(options(false), TQueryGlobals {});
+ context.state = &non_strict;
+ auto nullable = ColumnNullable::create(strings({"abcd"}),
ColumnUInt8::create(1, 0));
+ auto tuple = ColumnStruct::create(Columns {std::move(nullable),
ints({7})});
+ ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({1}));
+ const auto& array_before = assert_cast<const ColumnArray&>(*column);
+ const auto& tuple_before = assert_cast<const
ColumnStruct&>(array_before.get_data());
+ const auto& nullable_before = assert_cast<const
ColumnNullable&>(tuple_before.get_column(0));
+ ColumnPtr original = column;
+ const auto* array_address = column.get();
+ const auto* tuple_address = &tuple_before;
+ const auto* nullable_address = &nullable_before;
+ const auto* offsets_address = array_before.get_offsets().data();
+ const auto* nulls_address = nullable_before.get_null_map_data().data();
+ auto sibling_before = snapshot(tuple_before.get_column(1));
+ auto tuple_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(std::make_shared<DataTypeString>(2,
TYPE_VARCHAR)),
+ std::make_shared<DataTypeInt32>()});
+
+ column = TEST_TRY(validate(column,
std::make_shared<DataTypeArray>(tuple_type)));
+
+ const auto& array = assert_cast<const ColumnArray&>(*column);
+ const auto& checked_tuple = assert_cast<const
ColumnStruct&>(array.get_data());
+ const auto& checked_nullable = assert_cast<const
ColumnNullable&>(checked_tuple.get_column(0));
+ EXPECT_NE(column.get(), array_address);
+ EXPECT_NE(&checked_tuple, tuple_address);
+ EXPECT_NE(&checked_nullable, nullable_address);
+ EXPECT_EQ(array.get_offsets().data(), offsets_address);
+ EXPECT_EQ(checked_nullable.get_null_map_data().data(), nulls_address);
+ EXPECT_EQ(checked_nullable.get_nested_column().get_data_at(0).to_string(),
"ab");
+ EXPECT_EQ(checked_nullable.get_nested_column().use_count(), 1);
+ expect_same_storage(sibling_before, checked_tuple.get_column(1));
+ EXPECT_EQ(filter_map, (std::vector<char> {0}));
+ EXPECT_EQ(nullable_before.get_nested_column().get_data_at(0).to_string(),
"abcd");
+ column->sanity_check();
+}
+
+TEST_F(TabletColumnValidatorTest, SharedReadOnlyLeavesKeepStorage) {
+ const auto check = [this](ColumnPtr original, const DataTypePtr& type) {
+ auto before = snapshot(*original);
+ auto result = TEST_TRY(validate(original, type));
+ EXPECT_EQ(result.get(), original.get());
+ expect_same_storage(before, *result);
+ EXPECT_EQ(original->use_count(), 2);
+ };
+ check(ints({7, 8}), std::make_shared<DataTypeInt32>());
+ check(strings({"valid"}), std::make_shared<DataTypeJsonb>());
+ auto values = ColumnDecimal32::create(0, 0);
+ values->get_data().push_back(Decimal32(1));
+ ColumnPtr original = std::move(values);
+ const auto* data = assert_cast<const
ColumnDecimal32&>(*original).get_data().data();
+ auto result =
+ TEST_TRY(validate(original,
std::make_shared<DataTypeDecimal<TYPE_DECIMAL32>>(1, 0)));
+ EXPECT_EQ(result.get(), original.get());
+ EXPECT_EQ(assert_cast<const ColumnDecimal32&>(*result).get_data().data(),
data);
+}
+
+TEST_F(TabletColumnValidatorTest, SharedComplexTreeReusesReadOnlyLeaves) {
+ auto tuple = ColumnStruct::create(Columns {ints({7}), strings({"abc"})});
+ ColumnPtr original =
ColumnNullable::create(ColumnArray::create(std::move(tuple), offsets({1})),
+ ColumnUInt8::create(1, 0));
+ auto tuple_type = std::make_shared<DataTypeStruct>(
+ DataTypes {std::make_shared<DataTypeInt32>(),
std::make_shared<DataTypeString>()});
+ auto before = snapshot(*original);
+ auto result = TEST_TRY(
+ validate(original,
make_nullable(std::make_shared<DataTypeArray>(tuple_type))));
+ EXPECT_NE(result.get(), original.get());
+ expect_same_storage(before, *result);
+ expect_same_storage(before, *original);
+}
+
+} // namespace doris
diff --git
a/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out
b/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out
new file mode 100644
index 00000000000..5303e0253aa
--- /dev/null
+++ b/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !shared_column --
+401 {1:"abc"} {1:"abcdef"}
+402 {2:"uvw"} {2:"uvwxyz"}
diff --git
a/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy
b/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy
new file mode 100644
index 00000000000..1623ecb7678
--- /dev/null
+++
b/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_map_insert_shared_column", "p0") {
+ sql "drop table if exists test_map_insert_shared_column_src"
+ sql "drop table if exists test_map_insert_shared_column_dst"
+
+ sql """
+ create table test_map_insert_shared_column_src (
+ id int,
+ m map<int, string>
+ ) duplicate key(id)
+ distributed by hash(id) buckets 1
+ properties("replication_num" = "1")
+ """
+ sql """
+ create table test_map_insert_shared_column_dst (
+ id int,
+ m_short map<int, varchar(3)>,
+ m_full map<int, string>
+ ) duplicate key(id)
+ distributed by hash(id) buckets 1
+ properties("replication_num" = "1")
+ """
+ sql """
+ insert into test_map_insert_shared_column_src values
+ (401, {1: "abcdef"}),
+ (402, {2: "uvwxyz"})
+ """
+
+ sql "set enable_insert_strict = false"
+ sql """
+ insert into test_map_insert_shared_column_dst
+ select id, m, m from test_map_insert_shared_column_src
+ """
+ order_qt_shared_column """
+ select id, m_short, m_full
+ from test_map_insert_shared_column_dst
+ order by id
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]