This is an automated email from the ASF dual-hosted git repository.
HappenLee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6b57f55e240 [fix](function) Reject invalid regex pattern read from a
column instead of returning NULL (#68359)
6b57f55e240 is described below
commit 6b57f55e2402e11f2dd74b3402a633030c809326
Author: Jerry Hu <[email protected]>
AuthorDate: Mon Oct 12 12:17:19 2026 +0800
[fix](function) Reject invalid regex pattern read from a column instead of
returning NULL (#68359)
### What problem does this PR solve?
Issue Number: None
Problem Summary:
`regexp_extract`, `regexp_extract_or_null`, `regexp_extract_all`,
`regexp_extract_all_array`, `regexp_replace` and `regexp_replace_one`
handled
an invalid regex pattern differently depending on where the pattern came
from.
A constant pattern is compiled in `open()` and the query fails with
`INVALID_ARGUMENT`. A pattern read from a column is compiled per row in
the
execute path, and the same compile failure was swallowed: the row was
turned
into NULL and only a warning was recorded.
```sql
CREATE TABLE t (id INT, s STRING, p STRING, repl STRING) ...;
INSERT INTO t VALUES (1, 'abc', '[', 'x');
SELECT regexp_extract('abc', '[', 0); -- INVALID_ARGUMENT
SELECT regexp_extract(s, p, 0) FROM t; -- NULL
SELECT regexp_replace('abc', '[', 'x'); -- INVALID_ARGUMENT
SELECT regexp_replace(s, p, repl) FROM t; -- NULL
```
So the same malformed pattern was either rejected or silently produced
NULL
depending on the physical shape of the argument, which hides data
problems
behind an unrelated NULL. `regexp_count` and `regexp` already raised the
error
on the column path.
This PR makes the column path of the extract/replace families throw the
same
`Status::InvalidArgument` that the constant path returns from `open()`,
so an
invalid pattern always fails the query. The null-map plumbing that only
existed
to mark those rows is removed, together with the now-unused `push_null`
handlers of `regexp_extract_all`.
Raising an error on the column path must not swallow SQL NULL
propagation. The
framework's default NULL handling strips a Nullable argument to its
nested
column and executes every row, and the bytes under a NULL slot are
whatever
the producer left there, so `regexp_replace(s, concat(p, if(id = 2,
NULL, '')), repl)`
could still carry `[` under a NULL pattern. The regexp functions
therefore
handle NULL themselves now: a shared helper strips Nullable from the
arguments,
ORs their null maps into the result null map and short-circuits a NULL
constant
argument, and the execute loops skip NULL rows. No bytes under a NULL
slot are
ever compiled and no argument column is copied.
The shared helper chooses the NULL handling once per block. When no
argument
is Nullable (`have_null_column()`), the original columns go straight to
the
function with no input NULL map and the per-row NULL checks compiled out
(`if constexpr (CheckNull)`); only Nullable arguments take the unnesting
and
NULL-row skipping path. Input NULL checking is kept separate from output
nullability: `regexp_extract_or_null` still returns NULL on no match for
non-nullable input, and a nullable `regexp_count` result still gets its
wrapper. `regexp_count`, the replace family and the extract family all
go
through this one dispatch.
### Release note
`regexp_extract`, `regexp_extract_or_null`, `regexp_extract_all`,
`regexp_extract_all_array`, `regexp_replace` and `regexp_replace_one`
now fail
the query with `INVALID_ARGUMENT` when a regex pattern read from a
column does
not compile, matching the behavior for constant patterns. Previously
such rows
returned NULL.
### Check List (For Author)
- Test:
- Unit Test: `FunctionLikeTest.regexp_invalid_column_pattern_fails`
covers
every family with a mixed valid/invalid pattern column;
`FunctionLikeTest.regexp_null_pattern_hides_invalid_payload` covers a
nullable pattern column whose NULL row (placed between valid rows)
hides an invalid payload, a NULL constant pattern, and a NULL input
string with all-constant parameters;
`FunctionLikeTest.regexp_count_non_nullable_arguments` and
`FunctionLikeTest.regexp_non_nullable_arguments` cover non-nullable
arguments with column and constant patterns, including the NULL that
`regexp_extract_or_null` returns on no match; the existing
`FunctionLikeTest.*` and `function_string_test.*` still pass.
- Regression test: new `test_regexp_invalid_pattern` compares the
constant
and column paths for all families, checks NULL propagation with an
invalid payload under a NULL pattern between valid rows, and runs every
family over a NOT NULL table; `test_string_function_regexp`,
`test_regexp_count`, `test_regexp_replace`,
`test_regexp_repeated_search_anchor`, `test_regexp_chinese`,
`test_split_by_regexp`, `nereids_function_p0` `R` and
`fold_constant_nullable` pass locally.
- Behavior changed: Yes. An invalid pattern read from a column now fails
the
query instead of yielding NULL for that row.
- Does this need documentation: No
https://claude.ai/code/session_01CvzmDA8snJbHHhErBB58sB
https://claude.ai/code/session_01PqyWdaYLRvhohSJqcn14zH
---
be/src/exprs/function/function_regexp.cpp | 371 +++++++++++------
be/test/exprs/function/function_like_test.cpp | 458 +++++++++++++++++++++
.../test_regexp_invalid_pattern.out | 98 +++++
.../test_regexp_invalid_pattern.groovy | 173 ++++++++
4 files changed, 984 insertions(+), 116 deletions(-)
diff --git a/be/src/exprs/function/function_regexp.cpp
b/be/src/exprs/function/function_regexp.cpp
index a791e975c66..e4293c15b52 100644
--- a/be/src/exprs/function/function_regexp.cpp
+++ b/be/src/exprs/function/function_regexp.cpp
@@ -21,7 +21,9 @@
#include <stddef.h>
#include <unicode/utf8.h>
+#include <algorithm>
#include <boost/regex.hpp>
+#include <initializer_list>
#include <memory>
#include <string>
#include <string_view>
@@ -29,6 +31,7 @@
#include <utility>
#include <vector>
+#include "common/check.h"
#include "common/status.h"
#include "core/block/block.h"
#include "core/block/column_numbers.h"
@@ -45,9 +48,11 @@
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
+#include "core/field.h"
#include "core/string_ref.h"
#include "core/types.h"
#include "exec/common/stringop_substring.h"
+#include "exec/common/util.hpp"
#include "exprs/aggregate/aggregate_function.h"
#include "exprs/function/function.h"
#include "exprs/function/simple_function_factory.h"
@@ -75,6 +80,102 @@ static bool advance_re2_search_position(const char* data,
size_t size,
return true;
}
+// The regexp functions below handle NULL rows themselves
(use_default_implementation_for_nulls()
+// returns false). The framework's default path runs a function over the
nested column of a
+// Nullable argument, and the bytes stored under a NULL slot are whatever the
producer left there;
+// compiling them as a pattern could fail and abort a query whose result for
that row is simply
+// NULL.
+//
+// Strips Nullable from every argument into `nested_block` (a ColumnConst
wrapper stays, so the
+// const/full handling of the functions applies unchanged) and ORs the
argument null maps into
+// `null_map`, which the functions skip while executing and hand back as the
result null map.
+// Returns false when an argument is a NULL constant: the whole result is NULL.
+bool unnest_regexp_arguments(const Block& block, const ColumnNumbers&
arguments,
+ Block& nested_block, ColumnNumbers&
nested_arguments,
+ NullMap& null_map) {
+ for (const auto argument : arguments) {
+ const auto& column = block.get_by_position(argument);
+ NullableColumnInfo info;
+ if (column.type->is_nullable()) {
+ info = column.get_nullable_column_info();
+ if (info.only_null) {
+ return false;
+ }
+ if (info.has_null) {
+ // A ColumnConst holding NULL is only_null, so this is a full
column.
+ DCHECK(!info.is_const);
+ VectorizedUtils::update_null_map(null_map,
+
column.get_nullable_null_map_column()->get_data());
+ }
+ }
+ nested_arguments.push_back(nested_block.columns());
+ nested_block.insert(column.unnest_nullable(info, false));
+ }
+ return true;
+}
+
+// Shared execute() of the regexp functions, choosing the NULL handling once
per block. The
+// result column is `execute(check_null, block, arguments, null_map)`:
+// - no argument is Nullable: `check_null` is std::false_type,
`block`/`arguments` are the
+// original ones and `null_map` is nullptr, so the functions have no input
NULL row to skip;
+// - otherwise a NULL constant argument makes the result a NULL constant, else
`check_null` is
+// std::true_type, `block`/`arguments` are the Nullable-stripped arguments
and `null_map` marks
+// the NULL rows to skip.
+// Use regexp_result_null_map() to get the NULL map of a Nullable result.
+template <typename Execute>
+Status execute_regexp_with_nulls(Block& block, const ColumnNumbers& arguments,
uint32_t result,
+ size_t input_rows_count, Execute&& execute) {
+ auto& result_column = block.get_by_position(result);
+ if (!have_null_column(block, arguments)) {
+ result_column.column = execute(std::false_type {}, block, arguments,
nullptr);
+ return Status::OK();
+ }
+ auto null_map = ColumnUInt8::create(input_rows_count, 0);
+ Block nested_block;
+ ColumnNumbers nested_arguments;
+ if (!unnest_regexp_arguments(block, arguments, nested_block,
nested_arguments,
+ null_map->get_data())) {
+ result_column.column = result_column.type->create_column_const(
+ input_rows_count, Field::create_field<TYPE_NULL>(Null()));
+ return Status::OK();
+ }
+ result_column.column =
+ execute(std::true_type {}, nested_block, nested_arguments,
std::move(null_map));
+ return Status::OK();
+}
+
+// The NULL map of a regexp function's Nullable result: starts as the input
NULL rows handed out
+// by execute_regexp_with_nulls(), or all zeros when there is none. Functions
may still mark
+// further rows NULL, e.g. regexp_extract_or_null on no match.
+template <bool CheckNull>
+ColumnUInt8::MutablePtr regexp_result_null_map(ColumnUInt8::MutablePtr
null_map,
+ size_t input_rows_count) {
+ if constexpr (CheckNull) {
+ DCHECK(null_map);
+ return null_map;
+ } else {
+ DCHECK(!null_map);
+ return ColumnUInt8::create(input_rows_count, 0);
+ }
+}
+
+// Collects the argument columns of a regexp function whose first argument is
the input string
+// and `parameters` are the pattern-side arguments: the input string is
materialized, and the
+// parameters stay unpacked constants when all of them are constant, which is
returned.
+template <size_t N>
+bool prepare_regexp_argument_columns(Block& block, const ColumnNumbers&
arguments,
+ const std::initializer_list<size_t>&
parameters,
+ ColumnPtr (&argument_columns)[N]) {
+ bool col_const[N];
+ for (size_t i = 0; i < N; ++i) {
+ col_const[i] =
is_column_const(*block.get_by_position(arguments[i]).column);
+ }
+ argument_columns[0] =
+
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+ default_preprocess_parameter_columns(argument_columns, col_const,
parameters, block, arguments);
+ return std::ranges::all_of(parameters, [&](size_t i) { return
col_const[i]; });
+}
+
// Helper structure to hold either RE2 or Boost.Regex
struct RegexpExtractEngine {
std::unique_ptr<re2::RE2> re2_regex;
@@ -211,11 +312,20 @@ struct RegexpExtractEngine {
struct RegexpCountImpl {
using StringColumnView = ColumnView<TYPE_STRING>;
+ // `null_map` holds the NULL rows to skip when CheckNull, and is nullptr
otherwise.
+ template <bool CheckNull>
static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
- size_t input_rows_count, ColumnInt32::Container&
result_data) {
+ size_t input_rows_count, ColumnInt32::Container&
result_data,
+ const NullMap* null_map) {
auto str_col = StringColumnView::create(argument_columns[0]);
auto pattern_col = StringColumnView::create(argument_columns[1]);
for (size_t i = 0; i < input_rows_count; ++i) {
+ if constexpr (CheckNull) {
+ if ((*null_map)[i]) {
+ result_data[i] = 0;
+ continue;
+ }
+ }
DCHECK(!str_col.is_null_at(i));
DCHECK(!pattern_col.is_null_at(i));
result_data[i] = _execute_inner_loop(context, str_col,
pattern_col, i);
@@ -223,7 +333,7 @@ struct RegexpCountImpl {
}
static int _execute_inner_loop(FunctionContext* context, const
StringColumnView& str_col,
const StringColumnView& pattern_col, const
size_t index_now) {
- re2::RE2* re = reinterpret_cast<re2::RE2*>(
+ auto* re = reinterpret_cast<re2::RE2*>(
context->get_function_state(FunctionContext::THREAD_LOCAL));
std::unique_ptr<re2::RE2> scoped_re;
if (re == nullptr) {
@@ -234,7 +344,6 @@ struct RegexpCountImpl {
if (!st) {
context->add_warning(error_str.c_str());
throw Exception(Status::InvalidArgument(error_str));
- return 0;
}
re = scoped_re.get();
}
@@ -278,6 +387,8 @@ public:
return std::make_shared<DataTypeInt32>();
}
+ bool use_default_implementation_for_nulls() const override { return false;
}
+
Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
if (scope == FunctionContext::THREAD_LOCAL) {
if (context->is_col_constant(1)) {
@@ -305,17 +416,32 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
uint32_t result, size_t input_rows_count) const
override {
- auto result_data_column = ColumnInt32::create(input_rows_count);
- auto& result_data = result_data_column->get_data();
-
- ColumnPtr argument_columns[2];
-
- argument_columns[0] = block.get_by_position(arguments[0]).column;
- argument_columns[1] = block.get_by_position(arguments[1]).column;
- RegexpCountImpl::execute_impl(context, argument_columns,
input_rows_count, result_data);
-
- block.get_by_position(result).column = std::move(result_data_column);
- return Status::OK();
+ const bool result_nullable =
block.get_by_position(result).type->is_nullable();
+ return execute_regexp_with_nulls(
+ block, arguments, result, input_rows_count,
+ [&](auto check_null, const Block& args_block, const
ColumnNumbers& args,
+ ColumnUInt8::MutablePtr null_map) -> ColumnPtr {
+ auto result_data_column =
ColumnInt32::create(input_rows_count);
+ ColumnPtr argument_columns[2] =
{args_block.get_by_position(args[0]).column,
+
args_block.get_by_position(args[1]).column};
+ const NullMap* input_null_map = nullptr;
+ if constexpr (check_null) {
+ input_null_map = &null_map->get_data();
+ }
+ RegexpCountImpl::execute_impl<check_null>(
+ context, argument_columns, input_rows_count,
+ result_data_column->get_data(), input_null_map);
+ if constexpr (check_null) {
+ // The result of regexp_count is nullable whenever an
argument is.
+ DORIS_CHECK(result_nullable);
+ }
+ if (!result_nullable) {
+ return result_data_column;
+ }
+ return
ColumnNullable::create(std::move(result_data_column),
+
regexp_result_null_map<check_null>(
+ std::move(null_map),
input_rows_count));
+ });
}
};
@@ -353,6 +479,8 @@ public:
return make_nullable(std::make_shared<DataTypeString>());
}
+ bool use_default_implementation_for_nulls() const override { return false;
}
+
DataTypes get_variadic_argument_types_impl() const override {
return ParamTypes::get_variadic_argument_types();
}
@@ -392,43 +520,40 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
uint32_t result, size_t input_rows_count) const
override {
- size_t argument_size = arguments.size();
-
- auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
- auto result_data_column = ColumnString::create();
- auto& result_data = result_data_column->get_chars();
- auto& result_offset = result_data_column->get_offsets();
- result_offset.resize(input_rows_count);
-
- bool col_const[3];
- ColumnPtr argument_columns[3];
- for (int i = 0; i < 3; ++i) {
- col_const[i] =
is_column_const(*block.get_by_position(arguments[i]).column);
- }
- argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
-
*block.get_by_position(arguments[0]).column)
- .convert_to_full_column()
- :
block.get_by_position(arguments[0]).column;
-
- default_preprocess_parameter_columns(argument_columns, col_const, {1,
2}, block, arguments);
-
- StringRef options_value;
- if (col_const[1] && col_const[2]) {
- Impl::execute_impl_const_args(context, argument_columns,
options_value,
- input_rows_count, result_data,
result_offset,
- result_null_map->get_data());
- } else {
- // the options have check in FE, so is always const, and get idx
of 0
- if (argument_size == 4) {
- options_value =
block.get_by_position(arguments[3]).column->get_data_at(0);
- }
- Impl::execute_impl(context, argument_columns, options_value,
input_rows_count,
- result_data, result_offset,
result_null_map->get_data());
- }
+ return execute_regexp_with_nulls(
+ block, arguments, result, input_rows_count,
+ [&](auto check_null, Block& args_block, const ColumnNumbers&
args,
+ ColumnUInt8::MutablePtr null_map) -> ColumnPtr {
+ auto result_data_column = ColumnString::create();
+ auto& result_data = result_data_column->get_chars();
+ auto& result_offset = result_data_column->get_offsets();
+ result_offset.resize(input_rows_count);
+ auto result_null_map =
regexp_result_null_map<check_null>(std::move(null_map),
+
input_rows_count);
+
+ ColumnPtr argument_columns[3];
+ const bool parameters_const =
prepare_regexp_argument_columns(
+ args_block, args, {1, 2}, argument_columns);
+
+ StringRef options_value;
+ if (parameters_const) {
+ Impl::template execute_impl_const_args<check_null>(
+ context, argument_columns, options_value,
input_rows_count,
+ result_data, result_offset,
result_null_map->get_data());
+ } else {
+ // the options have check in FE, so is always const,
and get idx of 0
+ if (args.size() == 4) {
+ options_value =
+
args_block.get_by_position(args[3]).column->get_data_at(0);
+ }
+ Impl::template execute_impl<check_null>(
+ context, argument_columns, options_value,
input_rows_count,
+ result_data, result_offset,
result_null_map->get_data());
+ }
- block.get_by_position(result).column =
- ColumnNullable::create(std::move(result_data_column),
std::move(result_null_map));
- return Status::OK();
+ return
ColumnNullable::create(std::move(result_data_column),
+ std::move(result_null_map));
+ });
}
};
@@ -436,31 +561,47 @@ template <bool ReplaceOne>
struct RegexpReplaceImpl {
static constexpr auto name = ReplaceOne ? "regexp_replace_one" :
"regexp_replace";
+ // `null_map` holds the NULL rows to skip when CheckNull.
+ template <bool CheckNull>
static void execute_impl(FunctionContext* context, ColumnPtr
argument_columns[],
const StringRef& options_value, size_t
input_rows_count,
ColumnString::Chars& result_data,
ColumnString::Offsets& result_offset,
- NullMap& null_map) {
+ const NullMap& null_map) {
const auto* str_col =
check_and_get_column<ColumnString>(argument_columns[0].get());
const auto* pattern_col =
check_and_get_column<ColumnString>(argument_columns[1].get());
const auto* replace_col =
check_and_get_column<ColumnString>(argument_columns[2].get());
for (size_t i = 0; i < input_rows_count; ++i) {
+ if constexpr (CheckNull) {
+ if (null_map[i]) {
+ StringOP::push_empty_string(i, result_data, result_offset);
+ continue;
+ }
+ }
_execute_inner_loop<false>(context, str_col, pattern_col,
replace_col, options_value,
- result_data, result_offset, null_map,
i);
+ result_data, result_offset, i);
}
}
+ template <bool CheckNull>
static void execute_impl_const_args(FunctionContext* context, ColumnPtr
argument_columns[],
const StringRef& options_value, size_t
input_rows_count,
ColumnString::Chars& result_data,
- ColumnString::Offsets& result_offset,
NullMap& null_map) {
+ ColumnString::Offsets& result_offset,
+ const NullMap& null_map) {
const auto* str_col =
check_and_get_column<ColumnString>(argument_columns[0].get());
const auto* pattern_col =
check_and_get_column<ColumnString>(argument_columns[1].get());
const auto* replace_col =
check_and_get_column<ColumnString>(argument_columns[2].get());
for (size_t i = 0; i < input_rows_count; ++i) {
+ if constexpr (CheckNull) {
+ if (null_map[i]) {
+ StringOP::push_empty_string(i, result_data, result_offset);
+ continue;
+ }
+ }
_execute_inner_loop<true>(context, str_col, pattern_col,
replace_col, options_value,
- result_data, result_offset, null_map, i);
+ result_data, result_offset, i);
}
}
@@ -469,9 +610,8 @@ struct RegexpReplaceImpl {
const ColumnString* pattern_col,
const ColumnString* replace_col, const
StringRef& options_value,
ColumnString::Chars& result_data,
- ColumnString::Offsets& result_offset,
NullMap& null_map,
- const size_t index_now) {
- re2::RE2* re = reinterpret_cast<re2::RE2*>(
+ ColumnString::Offsets& result_offset,
const size_t index_now) {
+ auto* re = reinterpret_cast<re2::RE2*>(
context->get_function_state(FunctionContext::THREAD_LOCAL));
std::unique_ptr<re2::RE2> scoped_re; // destroys re if state->re is
nullptr
if (re == nullptr) {
@@ -481,8 +621,7 @@ struct RegexpReplaceImpl {
options_value, scoped_re);
if (!st) {
context->add_warning(error_str.c_str());
- StringOP::push_null_string(index_now, result_data,
result_offset, null_map);
- return;
+ throw Exception(Status::InvalidArgument(error_str));
}
re = scoped_re.get();
}
@@ -508,42 +647,36 @@ struct RegexpExtractImpl {
static DataTypePtr return_type() { return
make_nullable(std::make_shared<DataTypeString>()); }
- static Status execute(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
- uint32_t result, size_t input_rows_count) {
- bool col_const[3];
+ // See execute_regexp_with_nulls() for the arguments.
+ template <bool CheckNull>
+ static ColumnPtr execute(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+ size_t input_rows_count, ColumnUInt8::MutablePtr
input_null_map) {
ColumnPtr argument_columns[3];
- for (int i = 0; i < 3; ++i) {
- col_const[i] =
is_column_const(*block.get_by_position(arguments[i]).column);
- }
- argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
-
*block.get_by_position(arguments[0]).column)
- .convert_to_full_column()
- :
block.get_by_position(arguments[0]).column;
+ const bool parameters_const =
+ prepare_regexp_argument_columns(block, arguments, {1, 2},
argument_columns);
- auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
auto result_data_column = ColumnString::create();
auto& result_data = result_data_column->get_chars();
auto& result_offset = result_data_column->get_offsets();
result_offset.resize(input_rows_count);
+ auto result_null_map =
+ regexp_result_null_map<CheckNull>(std::move(input_null_map),
input_rows_count);
auto& null_map = result_null_map->get_data();
- default_preprocess_parameter_columns(argument_columns, col_const, {1,
2}, block, arguments);
-
- if (col_const[1] && col_const[2]) {
- _execute_loop<true>(context, argument_columns, input_rows_count,
result_data,
- result_offset, null_map);
+ if (parameters_const) {
+ _execute_loop<true, CheckNull>(context, argument_columns,
input_rows_count, result_data,
+ result_offset, null_map);
} else {
- _execute_loop<false>(context, argument_columns, input_rows_count,
result_data,
- result_offset, null_map);
+ _execute_loop<false, CheckNull>(context, argument_columns,
input_rows_count,
+ result_data, result_offset,
null_map);
}
- block.get_by_position(result).column =
- ColumnNullable::create(std::move(result_data_column),
std::move(result_null_map));
- return Status::OK();
+ return ColumnNullable::create(std::move(result_data_column),
std::move(result_null_map));
}
private:
- template <bool Const>
+ // `null_map` is the result NULL map; when CheckNull it starts with the
NULL rows to skip.
+ template <bool Const, bool CheckNull>
static void _execute_loop(FunctionContext* context, ColumnPtr
argument_columns[],
size_t input_rows_count, ColumnString::Chars&
result_data,
ColumnString::Offsets& result_offset, NullMap&
null_map) {
@@ -561,11 +694,23 @@ private:
return;
}
for (size_t i = 0; i < input_rows_count; ++i) {
+ if constexpr (CheckNull) {
+ if (null_map[i]) {
+ StringOP::push_empty_string(i, result_data,
result_offset);
+ continue;
+ }
+ }
_execute_inner_loop<true>(context, str_col, pattern_col,
index_data, result_data,
result_offset, null_map, i);
}
} else {
for (size_t i = 0; i < input_rows_count; ++i) {
+ if constexpr (CheckNull) {
+ if (null_map[i]) {
+ StringOP::push_empty_string(i, result_data,
result_offset);
+ continue;
+ }
+ }
const auto& index_data = index_col->get_int(i);
if (index_data < 0) {
ReturnNull ? StringOP::push_null_string(i, result_data,
result_offset, null_map)
@@ -596,8 +741,7 @@ private:
context->state()->enable_extended_regex());
if (!st) {
context->add_warning(error_str.c_str());
- StringOP::push_null_string(index_now, result_data,
result_offset, null_map);
- return;
+ throw Exception(Status::InvalidArgument(error_str));
}
engine = scoped_engine.get();
}
@@ -637,9 +781,6 @@ struct RegexpExtractAllStringOutput {
void push_empty(size_t index) {
StringOP::push_empty_string(index, result_data, result_offset);
}
- void push_null(size_t index, NullMap& null_map) {
- StringOP::push_null_string(index, result_data, result_offset,
null_map);
- }
void push_matches(size_t index, const std::vector<std::string>& matches) {
size_t total_size = 2; // '[' and ']'
for (const auto& m : matches) {
@@ -696,10 +837,6 @@ struct RegexpExtractAllArrayOutput {
UInt64 current_offset = 0;
void push_empty(size_t index) { array_offsets.push_back(current_offset); }
- void push_null(size_t index, NullMap& null_map) {
- null_map[index] = 1;
- array_offsets.push_back(current_offset);
- }
void push_matches(size_t index, const std::vector<std::string>& matches) {
for (const auto& m : matches) {
nested_col.insert_data(m.data(), m.size());
@@ -738,51 +875,46 @@ struct RegexpExtractAllImpl {
static DataTypePtr return_type() { return Handler::return_type(); }
- static Status execute(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
- uint32_t result, size_t input_rows_count) {
- bool col_const[2];
+ // See execute_regexp_with_nulls() for the arguments.
+ template <bool CheckNull>
+ static ColumnPtr execute(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+ size_t input_rows_count, ColumnUInt8::MutablePtr
input_null_map) {
ColumnPtr argument_columns[2];
- for (int i = 0; i < 2; ++i) {
- col_const[i] =
is_column_const(*block.get_by_position(arguments[i]).column);
- }
- argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
-
*block.get_by_position(arguments[0]).column)
- .convert_to_full_column()
- :
block.get_by_position(arguments[0]).column;
-
- default_preprocess_parameter_columns(argument_columns, col_const, {1},
block, arguments);
+ const bool pattern_const =
+ prepare_regexp_argument_columns(block, arguments, {1},
argument_columns);
const auto* str_col =
check_and_get_column<ColumnString>(argument_columns[0].get());
const auto* pattern_col =
check_and_get_column<ColumnString>(argument_columns[1].get());
- auto outer_null_map = ColumnUInt8::create(input_rows_count, 0);
- auto& null_map_data = outer_null_map->get_data();
-
+ auto result_null_map =
+ regexp_result_null_map<CheckNull>(std::move(input_null_map),
input_rows_count);
+ const auto& null_map = result_null_map->get_data();
typename Handler::State state(input_rows_count);
auto handler = state.create_handler();
std::visit(
[&](auto is_const) {
for (size_t i = 0; i < input_rows_count; ++i) {
- if (null_map_data[i]) {
- handler.push_null(i, null_map_data);
- continue;
+ if constexpr (CheckNull) {
+ if (null_map[i]) {
+ handler.push_empty(i);
+ continue;
+ }
}
regexp_extract_all_inner_loop<is_const>(context,
str_col, pattern_col,
- handler,
null_map_data, i);
+ handler, i);
}
},
- make_bool_variant(col_const[1]));
+ make_bool_variant(pattern_const));
- block.get_by_position(result).column =
state.finalize(std::move(outer_null_map));
- return Status::OK();
+ return state.finalize(std::move(result_null_map));
}
private:
template <bool is_const>
static void regexp_extract_all_inner_loop(FunctionContext* context, const
ColumnString* str_col,
const ColumnString* pattern_col,
Handler& handler,
- NullMap& null_map, const size_t
index_now) {
+ const size_t index_now) {
auto* engine = reinterpret_cast<RegexpExtractEngine*>(
context->get_function_state(FunctionContext::THREAD_LOCAL));
std::unique_ptr<RegexpExtractEngine> scoped_engine;
@@ -795,8 +927,7 @@ private:
context->state()->enable_extended_regex());
if (!st) {
context->add_warning(error_str.c_str());
- handler.push_null(index_now, null_map);
- return;
+ throw Exception(Status::InvalidArgument(error_str));
}
engine = scoped_engine.get();
}
@@ -833,6 +964,8 @@ public:
return Impl::return_type();
}
+ bool use_default_implementation_for_nulls() const override { return false;
}
+
Status open(FunctionContext* context, FunctionContext::FunctionStateScope
scope) override {
if (scope == FunctionContext::THREAD_LOCAL) {
if (context->is_col_constant(Impl::PATTERN_ARG_IDX)) {
@@ -861,7 +994,13 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
uint32_t result, size_t input_rows_count) const
override {
- return Impl::execute(context, block, arguments, result,
input_rows_count);
+ return execute_regexp_with_nulls(
+ block, arguments, result, input_rows_count,
+ [&](auto check_null, Block& args_block, const ColumnNumbers&
args,
+ ColumnUInt8::MutablePtr null_map) {
+ return Impl::template execute<check_null>(
+ context, args_block, args, input_rows_count,
std::move(null_map));
+ });
}
};
diff --git a/be/test/exprs/function/function_like_test.cpp
b/be/test/exprs/function/function_like_test.cpp
index 645b517e716..ec88452bbbe 100644
--- a/be/test/exprs/function/function_like_test.cpp
+++ b/be/test/exprs/function/function_like_test.cpp
@@ -17,6 +17,7 @@
#include <cstdint>
#include <memory>
+#include <optional>
#include <string>
#include <vector>
@@ -839,6 +840,463 @@ TEST(FunctionLikeTest, regexp_replace_one) {
}
}
+// An invalid pattern read from a column must fail the whole call with the same
+// InvalidArgument error a constant pattern raises in open(), instead of
turning
+// the offending row into NULL.
+// NOLINTNEXTLINE(readability-function-size) -- one driver covers every regexp
function family.
+TEST(FunctionLikeTest, regexp_invalid_column_pattern_fails) {
+ auto str_type = std::make_shared<DataTypeString>();
+ auto int_type = std::make_shared<DataTypeInt64>();
+
+ auto make_str_col = [](const std::vector<std::string>& values) {
+ auto col = ColumnString::create();
+ for (const auto& value : values) {
+ col->insert_data(value.data(), value.size());
+ }
+ return col;
+ };
+ auto make_int_col = [](const std::vector<int64_t>& values) {
+ auto col = ColumnInt64::create();
+ for (auto value : values) {
+ col->insert_value(value);
+ }
+ return col;
+ };
+
+ // The first row compiles, only the second row carries an invalid pattern.
+ const std::vector<std::string> strs = {"abc", "abc"};
+ const std::vector<std::string> patterns = {"(b)", "["};
+
+ auto run_case = [&](const std::string& func_name, ColumnsWithTypeAndName
arg_cols,
+ const DataTypePtr& return_type, const std::string&
expected_error) {
+ Block block;
+ ColumnNumbers arguments;
+ std::vector<DataTypePtr> arg_types;
+ std::vector<std::shared_ptr<ColumnPtrWrapper>> constant_cols;
+ for (auto& arg : arg_cols) {
+ arguments.push_back(static_cast<unsigned int>(block.columns()));
+ arg_types.push_back(arg.type);
+ constant_cols.push_back(nullptr);
+ block.insert(std::move(arg));
+ }
+ auto func = SimpleFunctionFactory::instance().get_function(
+ func_name, block.get_columns_with_type_and_name(),
return_type);
+ ASSERT_TRUE(func != nullptr) << func_name;
+
+ auto result = block.columns();
+ block.insert({nullptr, return_type, "result"});
+
+ FunctionUtils fn_utils({}, arg_types, false);
+ auto* fn_ctx = fn_utils.get_fn_ctx();
+ fn_ctx->set_constant_cols(constant_cols);
+
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::THREAD_LOCAL));
+
+ auto st = func->execute(fn_ctx, block, arguments, result, strs.size());
+ EXPECT_TRUE(st.is<ErrorCode::INVALID_ARGUMENT>()) << func_name << ": "
<< st;
+ EXPECT_NE(st.to_string().find(expected_error), std::string::npos)
+ << func_name << ": " << st;
+
+ static_cast<void>(func->close(fn_ctx, FunctionContext::THREAD_LOCAL));
+ static_cast<void>(func->close(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ };
+
+ auto nullable_str = make_nullable(str_type);
+ const std::string extract_error = "Invalid regex pattern";
+ const std::string re2_error = "Could not compile regexp pattern";
+
+ for (const char* func_name : {"regexp_extract", "regexp_extract_or_null"})
{
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_str_col(patterns), str_type, "pattern"},
+ {make_int_col({1, 1}), int_type, "idx"}},
+ nullable_str, extract_error);
+ }
+ run_case("regexp_extract_all",
+ {{make_str_col(strs), str_type, "str"}, {make_str_col(patterns),
str_type, "pattern"}},
+ nullable_str, extract_error);
+ run_case("regexp_extract_all_array",
+ {{make_str_col(strs), str_type, "str"}, {make_str_col(patterns),
str_type, "pattern"}},
+ make_nullable(std::make_shared<DataTypeArray>(nullable_str)),
extract_error);
+ for (const char* func_name : {"regexp_replace", "regexp_replace_one"}) {
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_str_col(patterns), str_type, "pattern"},
+ {make_str_col({"x", "x"}), str_type, "repl"}},
+ nullable_str, re2_error);
+ }
+ run_case("regexp_count",
+ {{make_str_col(strs), str_type, "str"}, {make_str_col(patterns),
str_type, "pattern"}},
+ std::make_shared<DataTypeInt32>(), re2_error);
+}
+
+// A SQL NULL pattern must yield NULL for that row even when the bytes stored
under the NULL
+// slot do not compile: the regexp functions skip NULL rows instead of running
the framework's
+// default NULL handling over the nested column. A NULL constant pattern
yields all NULLs, and a
+// NULL input string yields NULL for its row with constant parameters too.
+// NOLINTNEXTLINE(readability-function-size) -- one driver covers every regexp
function family.
+TEST(FunctionLikeTest, regexp_null_pattern_hides_invalid_payload) {
+ auto str_type = std::make_shared<DataTypeString>();
+ auto nullable_str = make_nullable(str_type);
+ auto int_type = std::make_shared<DataTypeInt64>();
+
+ auto make_str_col = [](const std::vector<std::string>& values) {
+ auto col = ColumnString::create();
+ for (const auto& value : values) {
+ col->insert_data(value.data(), value.size());
+ }
+ return col;
+ };
+ auto make_int_col = [](const std::vector<int64_t>& values) {
+ auto col = ColumnInt64::create();
+ for (auto value : values) {
+ col->insert_value(value);
+ }
+ return col;
+ };
+ auto make_null_map = [](const std::vector<uint8_t>& values) {
+ auto col = ColumnUInt8::create();
+ for (auto value : values) {
+ col->insert_value(value);
+ }
+ return col;
+ };
+ // The NULL row sits between two valid rows, so a skipped row must still
keep the result
+ // offsets of the rows after it right.
+ const std::vector<std::string> strs = {"abc", "abc", "abc"};
+ // Rows 0 and 2 are a valid pattern, row 1 is NULL with an invalid payload
underneath.
+ auto make_pattern_col = [&]() {
+ return ColumnNullable::create(make_str_col({"(b)", "[", "(b)"}),
make_null_map({0, 1, 0}));
+ };
+ // A NULL constant whose payload is an invalid pattern.
+ auto make_null_const_pattern = [&]() {
+ return ColumnConst::create(ColumnNullable::create(make_str_col({"["}),
make_null_map({1})),
+ strs.size());
+ };
+ // Row 1 is a NULL input string, for the case where every other parameter
is constant.
+ auto make_nullable_str_col = [&]() {
+ return ColumnNullable::create(make_str_col({"abc", "zzz", "abc"}),
+ make_null_map({0, 1, 0}));
+ };
+ auto make_const_str = [&](const std::string& value) {
+ return ColumnConst::create(make_str_col({value}), strs.size());
+ };
+ auto make_const_int = [&](int64_t value) {
+ return ColumnConst::create(make_int_col({value}), strs.size());
+ };
+
+ // Rows 0 and 2 are `expected`, std::nullopt meaning NULL; row 1 is always
NULL.
+ auto run_case = [&](const std::string& func_name, ColumnsWithTypeAndName
arg_cols,
+ const DataTypePtr& return_type,
+ const std::optional<std::string>& expected) {
+ Block block;
+ ColumnNumbers arguments;
+ std::vector<DataTypePtr> arg_types;
+ std::vector<std::shared_ptr<ColumnPtrWrapper>> constant_cols;
+ for (auto& arg : arg_cols) {
+ arguments.push_back(static_cast<unsigned int>(block.columns()));
+ arg_types.push_back(arg.type);
+ constant_cols.push_back(is_column_const(*arg.column)
+ ?
std::make_shared<ColumnPtrWrapper>(arg.column)
+ : nullptr);
+ block.insert(std::move(arg));
+ }
+ auto func = SimpleFunctionFactory::instance().get_function(
+ func_name, block.get_columns_with_type_and_name(),
return_type);
+ ASSERT_TRUE(func != nullptr) << func_name;
+
+ auto result = block.columns();
+ block.insert({nullptr, return_type, "result"});
+
+ FunctionUtils fn_utils({}, arg_types, false);
+ auto* fn_ctx = fn_utils.get_fn_ctx();
+ fn_ctx->set_constant_cols(constant_cols);
+
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::FRAGMENT_LOCAL)) << func_name;
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::THREAD_LOCAL)) << func_name;
+ ASSERT_EQ(Status::OK(), func->execute(fn_ctx, block, arguments,
result, strs.size()))
+ << func_name;
+
+ const auto& result_column = block.get_by_position(result).column;
+ ASSERT_EQ(strs.size(), result_column->size()) << func_name;
+ for (size_t row : {0, 2}) {
+ if (expected) {
+ EXPECT_EQ(*expected, return_type->to_string(*result_column,
row))
+ << func_name << " row " << row;
+ } else {
+ EXPECT_TRUE(result_column->is_null_at(row)) << func_name << "
row " << row;
+ }
+ }
+ EXPECT_TRUE(result_column->is_null_at(1)) << func_name;
+
+ static_cast<void>(func->close(fn_ctx, FunctionContext::THREAD_LOCAL));
+ static_cast<void>(func->close(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ };
+
+ for (const char* func_name : {"regexp_extract", "regexp_extract_or_null"})
{
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_pattern_col(), nullable_str, "pattern"},
+ {make_int_col({1, 1, 1}), int_type, "idx"}},
+ nullable_str, "b");
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_null_const_pattern(), nullable_str, "pattern"},
+ {make_int_col({1, 1, 1}), int_type, "idx"}},
+ nullable_str, std::nullopt);
+ run_case(func_name,
+ {{make_nullable_str_col(), nullable_str, "str"},
+ {make_const_str("(b)"), str_type, "pattern"},
+ {make_const_int(1), int_type, "idx"}},
+ nullable_str, "b");
+ }
+ run_case("regexp_extract_all",
+ {{make_str_col(strs), str_type, "str"}, {make_pattern_col(),
nullable_str, "pattern"}},
+ nullable_str, "['b']");
+ run_case("regexp_extract_all",
+ {{make_str_col(strs), str_type, "str"},
+ {make_null_const_pattern(), nullable_str, "pattern"}},
+ nullable_str, std::nullopt);
+ run_case("regexp_extract_all",
+ {{make_nullable_str_col(), nullable_str, "str"},
+ {make_const_str("(b)"), str_type, "pattern"}},
+ nullable_str, "['b']");
+ auto nullable_array =
make_nullable(std::make_shared<DataTypeArray>(nullable_str));
+ run_case("regexp_extract_all_array",
+ {{make_str_col(strs), str_type, "str"}, {make_pattern_col(),
nullable_str, "pattern"}},
+ nullable_array, "[\"b\"]");
+ run_case("regexp_extract_all_array",
+ {{make_str_col(strs), str_type, "str"},
+ {make_null_const_pattern(), nullable_str, "pattern"}},
+ nullable_array, std::nullopt);
+ run_case("regexp_extract_all_array",
+ {{make_nullable_str_col(), nullable_str, "str"},
+ {make_const_str("(b)"), str_type, "pattern"}},
+ nullable_array, "[\"b\"]");
+ for (const char* func_name : {"regexp_replace", "regexp_replace_one"}) {
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_pattern_col(), nullable_str, "pattern"},
+ {make_str_col({"x", "x", "x"}), str_type, "repl"}},
+ nullable_str, "axc");
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_null_const_pattern(), nullable_str, "pattern"},
+ {make_str_col({"x", "x", "x"}), str_type, "repl"}},
+ nullable_str, std::nullopt);
+ run_case(func_name,
+ {{make_nullable_str_col(), nullable_str, "str"},
+ {make_const_str("(b)"), str_type, "pattern"},
+ {make_const_str("x"), str_type, "repl"}},
+ nullable_str, "axc");
+ }
+ auto nullable_int32 = make_nullable(std::make_shared<DataTypeInt32>());
+ run_case("regexp_count",
+ {{make_str_col(strs), str_type, "str"}, {make_pattern_col(),
nullable_str, "pattern"}},
+ nullable_int32, "1");
+ run_case("regexp_count",
+ {{make_str_col(strs), str_type, "str"},
+ {make_null_const_pattern(), nullable_str, "pattern"}},
+ nullable_int32, std::nullopt);
+ run_case("regexp_count",
+ {{make_nullable_str_col(), nullable_str, "str"},
+ {make_const_str("(b)"), str_type, "pattern"}},
+ nullable_int32, "1");
+}
+
+// regexp_count over non-nullable arguments counts the original columns
directly and returns
+// the declared result type: a plain Int32 column, or a Nullable one without
NULL rows.
+TEST(FunctionLikeTest, regexp_count_non_nullable_arguments) {
+ auto str_type = std::make_shared<DataTypeString>();
+ auto int32_type = std::make_shared<DataTypeInt32>();
+
+ auto make_str_col = [](const std::vector<std::string>& values) {
+ auto col = ColumnString::create();
+ for (const auto& value : values) {
+ col->insert_data(value.data(), value.size());
+ }
+ return col;
+ };
+ const std::vector<std::string> strs = {"a1b22c333", "book keeper", "aaa"};
+
+ auto run_case = [&](ColumnsWithTypeAndName arg_cols, const DataTypePtr&
return_type,
+ const std::vector<int32_t>& expected) {
+ Block block;
+ ColumnNumbers arguments;
+ std::vector<DataTypePtr> arg_types;
+ std::vector<std::shared_ptr<ColumnPtrWrapper>> constant_cols;
+ for (auto& arg : arg_cols) {
+ arguments.push_back(static_cast<unsigned int>(block.columns()));
+ arg_types.push_back(arg.type);
+ constant_cols.push_back(is_column_const(*arg.column)
+ ?
std::make_shared<ColumnPtrWrapper>(arg.column)
+ : nullptr);
+ block.insert(std::move(arg));
+ }
+ auto func = SimpleFunctionFactory::instance().get_function(
+ "regexp_count", block.get_columns_with_type_and_name(),
return_type);
+ ASSERT_TRUE(func != nullptr);
+
+ auto result = block.columns();
+ block.insert({nullptr, return_type, "result"});
+
+ FunctionUtils fn_utils({}, arg_types, false);
+ auto* fn_ctx = fn_utils.get_fn_ctx();
+ fn_ctx->set_constant_cols(constant_cols);
+
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::THREAD_LOCAL));
+ ASSERT_EQ(Status::OK(), func->execute(fn_ctx, block, arguments,
result, strs.size()));
+
+ const auto& result_column = block.get_by_position(result).column;
+ ASSERT_EQ(return_type->is_nullable(), result_column->is_nullable());
+ const IColumn* data_column = result_column.get();
+ if (return_type->is_nullable()) {
+ const auto& nullable = assert_cast<const
ColumnNullable&>(*result_column);
+ EXPECT_FALSE(nullable.has_null());
+ data_column = &nullable.get_nested_column();
+ }
+ const auto& data = assert_cast<const
ColumnInt32&>(*data_column).get_data();
+ ASSERT_EQ(expected.size(), data.size());
+ for (size_t i = 0; i < expected.size(); ++i) {
+ EXPECT_EQ(expected[i], data[i]) << "row " << i;
+ }
+
+ static_cast<void>(func->close(fn_ctx, FunctionContext::THREAD_LOCAL));
+ static_cast<void>(func->close(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ };
+
+ for (const auto& return_type : {DataTypePtr(int32_type),
make_nullable(int32_type)}) {
+ run_case({{make_str_col(strs), str_type, "str"},
+ {make_str_col({"\\d+", "oo|ee", "^a"}), str_type,
"pattern"}},
+ return_type, {3, 2, 1});
+ run_case({{make_str_col(strs), str_type, "str"},
+ {ColumnConst::create(make_str_col({"[a-z]"}), strs.size()),
str_type, "pattern"}},
+ return_type, {3, 10, 3});
+ }
+}
+
+// The extract and replace functions over non-nullable arguments run without
an input NULL map
+// but keep their Nullable result: no row is NULL except where the function
itself produces one,
+// e.g. regexp_extract_or_null on no match.
+// NOLINTNEXTLINE(readability-function-size) -- one driver covers every regexp
function family.
+TEST(FunctionLikeTest, regexp_non_nullable_arguments) {
+ auto str_type = std::make_shared<DataTypeString>();
+ auto nullable_str = make_nullable(str_type);
+ auto int_type = std::make_shared<DataTypeInt64>();
+
+ auto make_str_col = [](const std::vector<std::string>& values) {
+ auto col = ColumnString::create();
+ for (const auto& value : values) {
+ col->insert_data(value.data(), value.size());
+ }
+ return col;
+ };
+ auto make_int_col = [](const std::vector<int64_t>& values) {
+ auto col = ColumnInt64::create();
+ for (auto value : values) {
+ col->insert_value(value);
+ }
+ return col;
+ };
+ // Row 0 matches, row 1 does not, for both the column and the constant
pattern.
+ const std::vector<std::string> strs = {"abc", "xyz"};
+ auto make_column_pattern = [&]() { return make_str_col({"(b)", "(q)"}); };
+ auto make_const_pattern = [&]() {
+ return ColumnConst::create(make_str_col({"(b)"}), strs.size());
+ };
+
+ // std::nullopt marks a NULL row.
+ using Expected = std::vector<std::optional<std::string>>;
+ auto run_case = [&](const std::string& func_name, ColumnsWithTypeAndName
arg_cols,
+ const DataTypePtr& return_type, const Expected&
expected) {
+ Block block;
+ ColumnNumbers arguments;
+ std::vector<DataTypePtr> arg_types;
+ std::vector<std::shared_ptr<ColumnPtrWrapper>> constant_cols;
+ for (auto& arg : arg_cols) {
+ arguments.push_back(static_cast<unsigned int>(block.columns()));
+ arg_types.push_back(arg.type);
+ constant_cols.push_back(is_column_const(*arg.column)
+ ?
std::make_shared<ColumnPtrWrapper>(arg.column)
+ : nullptr);
+ block.insert(std::move(arg));
+ }
+ auto func = SimpleFunctionFactory::instance().get_function(
+ func_name, block.get_columns_with_type_and_name(),
return_type);
+ ASSERT_TRUE(func != nullptr) << func_name;
+
+ auto result = block.columns();
+ block.insert({nullptr, return_type, "result"});
+
+ FunctionUtils fn_utils({}, arg_types, false);
+ auto* fn_ctx = fn_utils.get_fn_ctx();
+ fn_ctx->set_constant_cols(constant_cols);
+
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::FRAGMENT_LOCAL)) << func_name;
+ ASSERT_EQ(Status::OK(), func->open(fn_ctx,
FunctionContext::THREAD_LOCAL)) << func_name;
+ ASSERT_EQ(Status::OK(), func->execute(fn_ctx, block, arguments,
result, strs.size()))
+ << func_name;
+
+ const auto& result_column = block.get_by_position(result).column;
+ ASSERT_TRUE(result_column->is_nullable()) << func_name;
+ ASSERT_EQ(expected.size(), result_column->size()) << func_name;
+ for (size_t i = 0; i < expected.size(); ++i) {
+ if (expected[i]) {
+ EXPECT_FALSE(result_column->is_null_at(i)) << func_name << "
row " << i;
+ EXPECT_EQ(*expected[i], return_type->to_string(*result_column,
i))
+ << func_name << " row " << i;
+ } else {
+ EXPECT_TRUE(result_column->is_null_at(i)) << func_name << "
row " << i;
+ }
+ }
+
+ static_cast<void>(func->close(fn_ctx, FunctionContext::THREAD_LOCAL));
+ static_cast<void>(func->close(fn_ctx,
FunctionContext::FRAGMENT_LOCAL));
+ };
+
+ auto nullable_array =
make_nullable(std::make_shared<DataTypeArray>(nullable_str));
+ for (bool const_pattern : {false, true}) {
+ // With a constant pattern the other parameters are constant too,
which takes the
+ // functions' all-constant-parameters path.
+ auto make_pattern = [&]() -> ColumnPtr {
+ return const_pattern ? ColumnPtr(make_const_pattern()) :
make_column_pattern();
+ };
+ auto make_index = [&]() -> ColumnPtr {
+ return const_pattern ?
ColumnPtr(ColumnConst::create(make_int_col({1}), strs.size()))
+ : make_int_col({1, 1});
+ };
+ auto make_replacement = [&]() -> ColumnPtr {
+ return const_pattern ?
ColumnPtr(ColumnConst::create(make_str_col({"x"}), strs.size()))
+ : make_str_col({"x", "x"});
+ };
+ run_case("regexp_extract",
+ {{make_str_col(strs), str_type, "str"},
+ {make_pattern(), str_type, "pattern"},
+ {make_index(), int_type, "idx"}},
+ nullable_str, {"b", ""});
+ run_case("regexp_extract_or_null",
+ {{make_str_col(strs), str_type, "str"},
+ {make_pattern(), str_type, "pattern"},
+ {make_index(), int_type, "idx"}},
+ nullable_str, {"b", std::nullopt});
+ run_case("regexp_extract_all",
+ {{make_str_col(strs), str_type, "str"}, {make_pattern(),
str_type, "pattern"}},
+ nullable_str, {"['b']", ""});
+ run_case("regexp_extract_all_array",
+ {{make_str_col(strs), str_type, "str"}, {make_pattern(),
str_type, "pattern"}},
+ nullable_array, {"[\"b\"]", "[]"});
+ for (const char* func_name : {"regexp_replace", "regexp_replace_one"})
{
+ run_case(func_name,
+ {{make_str_col(strs), str_type, "str"},
+ {make_pattern(), str_type, "pattern"},
+ {make_replacement(), str_type, "repl"}},
+ nullable_str, {"axc", "xyz"});
+ }
+ }
+}
+
// Enhanced tests for better coverage
TEST(FunctionLikeTest, pattern_optimization_allpass) {
diff --git
a/regression-test/data/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.out
b/regression-test/data/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.out
new file mode 100644
index 00000000000..1f8a7f56d8c
--- /dev/null
+++
b/regression-test/data/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.out
@@ -0,0 +1,98 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !extract_valid --
+1 b
+3 \N
+4 b
+
+-- !extract_or_null_valid --
+1 b
+3 \N
+4 b
+
+-- !extract_all_valid --
+1 ['b']
+3 \N
+4 ['b']
+
+-- !extract_all_array_valid --
+1 ["b"]
+3 \N
+4 ["b"]
+
+-- !replace_valid --
+1 axc
+3 \N
+4 axc
+
+-- !replace_one_valid --
+1 axc
+3 \N
+4 axc
+
+-- !count_valid --
+1 1
+3 \N
+4 1
+
+-- !extract_null_payload --
+1 b
+2 \N
+3 \N
+4 b
+
+-- !extract_or_null_null_payload --
+1 b
+2 \N
+3 \N
+4 b
+
+-- !extract_all_null_payload --
+1 ['b']
+2 \N
+3 \N
+4 ['b']
+
+-- !extract_all_array_null_payload --
+1 ["b"]
+2 \N
+3 \N
+4 ["b"]
+
+-- !replace_null_payload --
+1 axc
+2 \N
+3 \N
+4 axc
+
+-- !replace_one_null_payload --
+1 axc
+2 \N
+3 \N
+4 axc
+
+-- !count_null_payload --
+1 1
+2 \N
+3 \N
+4 1
+
+-- !extract_null_const --
+1 \N \N
+
+-- !extract_all_null_const --
+1 \N \N
+
+-- !replace_null_const --
+1 \N \N
+
+-- !count_null_const --
+1 \N \N
+
+-- !not_null_column_pattern --
+1 b b ['b'] ["b"] axc axc 1
+2 \N [] xyz xyz 0
+
+-- !not_null_const_pattern --
+1 b b ['b'] ["b"] axc axc 1
+2 \N [] xyz xyz 0
+
diff --git
a/regression-test/suites/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.groovy
b/regression-test/suites/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.groovy
new file mode 100644
index 00000000000..e87cdbb1da2
--- /dev/null
+++
b/regression-test/suites/query_p0/sql_functions/string_functions/test_regexp_invalid_pattern.groovy
@@ -0,0 +1,173 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// An invalid regex pattern must raise the same error whether it is a constant
+// or read from a column; it must never be silently turned into NULL.
+suite("test_regexp_invalid_pattern") {
+ sql "DROP TABLE IF EXISTS test_regexp_invalid_pattern"
+ sql """
+ CREATE TABLE test_regexp_invalid_pattern (
+ id INT,
+ s STRING,
+ p STRING,
+ repl STRING
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1");
+ """
+ sql """
+ INSERT INTO test_regexp_invalid_pattern VALUES
+ (1, 'abc', '(b)', 'x'),
+ (2, 'abc', '[', 'x'),
+ (3, 'abc', NULL, 'x'),
+ (4, 'abc', '(b)', 'x');
+ """
+
+ // Valid column patterns keep working, a NULL pattern between them still
yields NULL.
+ order_qt_extract_valid "SELECT id, regexp_extract(s, p, 1) FROM
test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_extract_or_null_valid "SELECT id, regexp_extract_or_null(s, p, 1)
FROM test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_extract_all_valid "SELECT id, regexp_extract_all(s, p) FROM
test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_extract_all_array_valid "SELECT id, regexp_extract_all_array(s,
p) FROM test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_replace_valid "SELECT id, regexp_replace(s, p, repl) FROM
test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_replace_one_valid "SELECT id, regexp_replace_one(s, p, repl) FROM
test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+ order_qt_count_valid "SELECT id, regexp_count(s, p) FROM
test_regexp_invalid_pattern WHERE id IN (1, 3, 4)"
+
+ // Constant invalid pattern: rejected.
+ test {
+ sql "SELECT regexp_extract('abc', '[', 0)"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_or_null('abc', '[', 0)"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_all('abc', '[')"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_all_array('abc', '[')"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_replace('abc', '[', 'x')"
+ exception "Could not compile regexp pattern"
+ }
+ test {
+ sql "SELECT regexp_replace_one('abc', '[', 'x')"
+ exception "Could not compile regexp pattern"
+ }
+ test {
+ sql "SELECT regexp_count('abc', '[')"
+ exception "Could not compile regexp pattern"
+ }
+
+ // Column invalid pattern: rejected with the same error instead of
returning NULL.
+ test {
+ sql "SELECT regexp_extract(s, p, 0) FROM test_regexp_invalid_pattern"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_or_null(s, p, 0) FROM
test_regexp_invalid_pattern"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_all(s, p) FROM test_regexp_invalid_pattern"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_extract_all_array(s, p) FROM
test_regexp_invalid_pattern"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_replace(s, p, repl) FROM
test_regexp_invalid_pattern"
+ exception "Could not compile regexp pattern"
+ }
+ test {
+ sql "SELECT regexp_replace_one(s, p, repl) FROM
test_regexp_invalid_pattern"
+ exception "Could not compile regexp pattern"
+ }
+ test {
+ sql "SELECT regexp_count(s, p) FROM test_regexp_invalid_pattern"
+ exception "Could not compile regexp pattern"
+ }
+
+ // A SQL NULL pattern yields NULL for that row even when the bytes left
under the NULL
+ // slot (here the invalid '[' of row 2, kept by concat) do not compile.
+ order_qt_extract_null_payload "SELECT id, regexp_extract(s, concat(p,
if(id = 2, NULL, '')), 1) FROM test_regexp_invalid_pattern"
+ order_qt_extract_or_null_null_payload "SELECT id,
regexp_extract_or_null(s, concat(p, if(id = 2, NULL, '')), 1) FROM
test_regexp_invalid_pattern"
+ order_qt_extract_all_null_payload "SELECT id, regexp_extract_all(s,
concat(p, if(id = 2, NULL, ''))) FROM test_regexp_invalid_pattern"
+ order_qt_extract_all_array_null_payload "SELECT id,
regexp_extract_all_array(s, concat(p, if(id = 2, NULL, ''))) FROM
test_regexp_invalid_pattern"
+ order_qt_replace_null_payload "SELECT id, regexp_replace(s, concat(p,
if(id = 2, NULL, '')), repl) FROM test_regexp_invalid_pattern"
+ order_qt_replace_one_null_payload "SELECT id, regexp_replace_one(s,
concat(p, if(id = 2, NULL, '')), repl) FROM test_regexp_invalid_pattern"
+ order_qt_count_null_payload "SELECT id, regexp_count(s, concat(p, if(id =
2, NULL, ''))) FROM test_regexp_invalid_pattern"
+
+ // NULL constant pattern and NULL input string keep propagating NULL.
+ order_qt_extract_null_const "SELECT id, regexp_extract(s, NULL, 1),
regexp_extract(NULL, p, 1) FROM test_regexp_invalid_pattern WHERE id = 1"
+ order_qt_extract_all_null_const "SELECT id, regexp_extract_all(s, NULL),
regexp_extract_all_array(NULL, p) FROM test_regexp_invalid_pattern WHERE id = 1"
+ order_qt_replace_null_const "SELECT id, regexp_replace(s, NULL, repl),
regexp_replace_one(s, p, NULL) FROM test_regexp_invalid_pattern WHERE id = 1"
+ order_qt_count_null_const "SELECT id, regexp_count(s, NULL),
regexp_count(NULL, p) FROM test_regexp_invalid_pattern WHERE id = 1"
+
+ // Constant string with column pattern is a column path too.
+ test {
+ sql "SELECT regexp_extract('abc', p, 0) FROM
test_regexp_invalid_pattern WHERE id = 2"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_replace('abc', p, 'x') FROM
test_regexp_invalid_pattern WHERE id = 2"
+ exception "Could not compile regexp pattern"
+ }
+
+ // NOT NULL arguments run without an input NULL map: the results keep
their Nullable type,
+ // regexp_extract_or_null still returns NULL on no match, and an invalid
column pattern is
+ // still rejected.
+ sql "DROP TABLE IF EXISTS test_regexp_invalid_pattern_not_null"
+ sql """
+ CREATE TABLE test_regexp_invalid_pattern_not_null (
+ id INT NOT NULL,
+ s STRING NOT NULL,
+ p STRING NOT NULL,
+ repl STRING NOT NULL
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES("replication_num" = "1");
+ """
+ sql """
+ INSERT INTO test_regexp_invalid_pattern_not_null VALUES
+ (1, 'abc', '(b)', 'x'),
+ (2, 'xyz', '(q)', 'x');
+ """
+ order_qt_not_null_column_pattern """
+ SELECT id, regexp_extract(s, p, 1), regexp_extract_or_null(s, p, 1),
+ regexp_extract_all(s, p), regexp_extract_all_array(s, p),
+ regexp_replace(s, p, repl), regexp_replace_one(s, p, repl),
regexp_count(s, p)
+ FROM test_regexp_invalid_pattern_not_null
+ """
+ order_qt_not_null_const_pattern """
+ SELECT id, regexp_extract(s, '(b)', 1), regexp_extract_or_null(s,
'(b)', 1),
+ regexp_extract_all(s, '(b)'), regexp_extract_all_array(s,
'(b)'),
+ regexp_replace(s, '(b)', 'x'), regexp_replace_one(s, '(b)',
'x'),
+ regexp_count(s, '(b)')
+ FROM test_regexp_invalid_pattern_not_null
+ """
+ test {
+ sql "SELECT regexp_extract(s, concat(p, '['), 1) FROM
test_regexp_invalid_pattern_not_null"
+ exception "Invalid regex pattern"
+ }
+ test {
+ sql "SELECT regexp_replace(s, concat(p, '['), repl) FROM
test_regexp_invalid_pattern_not_null"
+ exception "Could not compile regexp pattern"
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
