This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d24fafc0b34 [fix](like) Fix LIKE matching values with extra characters
at the end (#68133)
d24fafc0b34 is described below
commit d24fafc0b343bf7a1b23c5d1addbb937e032c8fd
Author: Chenyang Sun <[email protected]>
AuthorDate: Fri Sep 18 13:48:42 2026 +0800
[fix](like) Fix LIKE matching values with extra characters at the end
(#68133)
### What problem does this PR solve?
1. `LIKE 'a_b'` is converted to `^a.b\z`, not `^a.b$`. Hyperscan follows
PCRE, where `$` also matches right before a newline that ends the value,
so `^a.b$` wrongly matched 'acb\n'. `\z` is the end of the value in both
Hyperscan and RE2.
2. `LIKE 'a_b%'` is converted to `^a.b`, not `^a.b.*\z`. A trailing `%`
matches anything, and so does appending nothing. The anchored form would
force `.*` to consume the value to its end, and `.*` only matches valid
UTF-8, so RE2 would not match 'acb\xff' while Hyperscan still would.
3. `LIKE 'a_*'` is converted to `^a.\*\z`, not `^a.\*`. The tail anchor
used to be skipped whenever the produced regex ended with `*`, a check
meant to detect the `.*` expanded from a trailing `%`. An escaped
literal `*` ends the regex with the same character, so the pattern lost
its anchor entirely and `'ab*xyz'` matched. The decision now comes from
the position in the pattern instead of from the produced string.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
be/src/exprs/function/like.cpp | 13 ++--
be/test/exprs/function/function_like_test.cpp | 74 ++++++++++++++++++++
.../test_like_whole_string_match.out | 74 ++++++++++++++++++++
.../test_like_whole_string_match.groovy | 81 ++++++++++++++++++++++
4 files changed, 237 insertions(+), 5 deletions(-)
diff --git a/be/src/exprs/function/like.cpp b/be/src/exprs/function/like.cpp
index 938ecfb958f..a6403f5684c 100644
--- a/be/src/exprs/function/like.cpp
+++ b/be/src/exprs/function/like.cpp
@@ -781,7 +781,8 @@ void FunctionLike::convert_like_pattern(const
LikeSearchState* state, const std:
re_pattern->clear();
if (pattern.empty()) {
- re_pattern->append("^$");
+ // `\z` is the end of the value in both Hyperscan and RE2
+ re_pattern->append("^\\z");
return;
}
@@ -809,6 +810,10 @@ void FunctionLike::convert_like_pattern(const
LikeSearchState* state, const std:
}
if (c == '%') {
+ if (i + 1 == pattern.size()) {
+ // a trailing `%` matches anything, and so does appending
nothing
+ return;
+ }
re_pattern->append(".*");
} else if (c == '_') {
re_pattern->append(".");
@@ -823,10 +828,8 @@ void FunctionLike::convert_like_pattern(const
LikeSearchState* state, const std:
}
}
- // add $ to pattern tail to match line tail
- if (!pattern.empty() && re_pattern->back() != '*') {
- re_pattern->append("$");
- }
+ // `\z` is the end of the value in both Hyperscan and RE2
+ re_pattern->append("\\z");
}
void FunctionLike::remove_escape_character(std::string* search_string) {
diff --git a/be/test/exprs/function/function_like_test.cpp
b/be/test/exprs/function/function_like_test.cpp
index 62384b3f82b..a6887222251 100644
--- a/be/test/exprs/function/function_like_test.cpp
+++ b/be/test/exprs/function/function_like_test.cpp
@@ -144,6 +144,80 @@ TEST(FunctionLikeTest, like) {
func_name, const_pattern_input_types, data_set));
}
+TEST(FunctionLikeTest, like_matches_whole_value) {
+ std::string func_name = "like";
+
+ DataSet data_set = {
+ // A trailing newline belongs to the value, so a pattern that is
anchored at the
+ // tail must not match across it.
+ {{std::string("acb"), std::string("a_b")}, uint8_t(1)},
+ {{std::string("acb\n"), std::string("a_b")}, uint8_t(0)},
+ {{std::string("acb\r\n"), std::string("a_b")}, uint8_t(0)},
+ {{std::string("acb\n\n"), std::string("a_b")}, uint8_t(0)},
+ {{std::string("acbx"), std::string("a_b")}, uint8_t(0)},
+ {{std::string("\nacb"), std::string("a_b")}, uint8_t(0)},
+ {{std::string("abc"), std::string("a%c")}, uint8_t(1)},
+ {{std::string("abc\n"), std::string("a%c")}, uint8_t(0)},
+ {{std::string("abc\n"), std::string("%b%c")}, uint8_t(0)},
+ // The newline is an ordinary character for '_' and '%'.
+ {{std::string("a\nb"), std::string("a_b")}, uint8_t(1)},
+ {{std::string("a\nb"), std::string("a%b")}, uint8_t(1)},
+ {{std::string("acb\n"), std::string("a_b_")}, uint8_t(1)},
+ {{std::string("acb\n"), std::string("a_b%")}, uint8_t(1)},
+ {{std::string("abc\n"), std::string("a_c%")}, uint8_t(1)},
+ // '_' stands for one character, not for one byte.
+ {{std::string("a中b"), std::string("a_b")}, uint8_t(1)},
+ {{std::string("a中b\n"), std::string("a_b")}, uint8_t(0)},
+ // An empty pattern only matches an empty value.
+ {{std::string(""), std::string("")}, uint8_t(1)},
+ {{std::string("\n"), std::string("")}, uint8_t(0)},
+ // The shortcut paths and the regex path must agree on the same
value.
+ {{std::string("acb\n"), std::string("acb")}, uint8_t(0)},
+ {{std::string("abc\n"), std::string("%c")}, uint8_t(0)},
+ {{std::string("abc\n"), std::string("a%")}, uint8_t(1)},
+
+ // A literal '*' at the tail of the pattern is not the '.*'
expanded from a
+ // trailing '%', so the pattern stays anchored.
+ {{std::string("ab*"), std::string("a_*")}, uint8_t(1)},
+ {{std::string("ab*xyz"), std::string("a_*")}, uint8_t(0)},
+ {{std::string("ab*\n"), std::string("a_*")}, uint8_t(0)},
+ {{std::string("ab%"), std::string("a_\\%")}, uint8_t(1)},
+ {{std::string("ab%xyz"), std::string("a_\\%")}, uint8_t(0)},
+ };
+
+ InputTypeSet const_pattern_input_types = {PrimitiveType::TYPE_VARCHAR,
+ PrimitiveType::TYPE_VARCHAR};
+ check_function_all_arg_comb<DataTypeUInt8, true>(func_name,
const_pattern_input_types,
+ data_set);
+}
+
+TEST(FunctionLikeTest, convert_like_pattern_shapes) {
+ auto convert = [](const std::string& pattern) {
+ std::string re_pattern;
+ FunctionLike::convert_like_pattern(nullptr, pattern, &re_pattern);
+ return re_pattern;
+ };
+
+ // The tail anchor is `\z`, never `$`: Hyperscan reads `$` the PCRE way
and would also
+ // match right before a newline that ends the value.
+ EXPECT_EQ(convert(""), "^\\z");
+ EXPECT_EQ(convert("a_b"), "^a.b\\z");
+ EXPECT_EQ(convert("%c%b"), ".*c.*b\\z");
+
+ // A trailing `%` is the only shape left open at the tail, and it needs no
`.*` either.
+ EXPECT_EQ(convert("abc%"), "^abc");
+ EXPECT_EQ(convert("a_b%%"), "^a.b.*");
+ EXPECT_EQ(convert("%"), "");
+
+ // An escaped literal `*` ends the produced regex with '*' without being a
wildcard, and an
+ // escaped `%` is a literal: both stay anchored.
+ EXPECT_EQ(convert("a_*"), "^a.\\*\\z");
+ EXPECT_EQ(convert("a_\\%"), "^a.%\\z");
+
+ // A backslash that does not open a LIKE escape is a literal backslash.
+ EXPECT_EQ(convert("a_\\"), "^a.\\\\\\z");
+}
+
TEST(FunctionLikeTest, regexp) {
std::string func_name = "regexp";
diff --git
a/regression-test/data/query_p0/sql_functions/string_functions/test_like_whole_string_match.out
b/regression-test/data/query_p0/sql_functions/string_functions/test_like_whole_string_match.out
new file mode 100644
index 00000000000..4dbfc4fbc89
--- /dev/null
+++
b/regression-test/data/query_p0/sql_functions/string_functions/test_like_whole_string_match.out
@@ -0,0 +1,74 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !anchor_underscore --
+1 3 true
+2 4 false
+3 5 false
+4 3 true
+5 4 false
+6 3 false
+7 4 false
+8 6 false
+
+-- !anchor_leading_percent --
+1 3 true
+2 4 false
+3 5 false
+4 3 false
+5 4 false
+6 3 false
+7 4 false
+8 6 false
+
+-- !literal_star_tail --
+1 3 false
+2 4 false
+3 5 false
+4 3 false
+5 4 false
+6 3 true
+7 4 false
+8 6 false
+
+-- !trailing_percent --
+1 3 true
+2 4 true
+3 5 true
+4 3 true
+5 4 true
+6 3 false
+7 4 false
+8 6 false
+
+-- !equals_shortcut --
+1 3 true
+2 4 false
+3 5 false
+4 3 false
+5 4 false
+6 3 false
+7 4 false
+8 6 false
+
+-- !pushdown_underscore --
+1
+4
+
+-- !pushdown_literal_star --
+6
+
+-- !not_literal_star --
+1
+2
+3
+4
+5
+7
+8
+
+-- !pushdown_trailing_percent --
+1
+2
+3
+4
+5
+
diff --git
a/regression-test/suites/query_p0/sql_functions/string_functions/test_like_whole_string_match.groovy
b/regression-test/suites/query_p0/sql_functions/string_functions/test_like_whole_string_match.groovy
new file mode 100644
index 00000000000..89a960e09fb
--- /dev/null
+++
b/regression-test/suites/query_p0/sql_functions/string_functions/test_like_whole_string_match.groovy
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_like_whole_string_match") {
+ sql "drop table if exists test_like_whole_string_match"
+ sql """
+ create table test_like_whole_string_match (
+ id int,
+ s varchar(64)
+ ) duplicate key(id)
+ distributed by hash(id) buckets 1
+ properties("replication_num" = "1");
+ """
+ sql """
+ insert into test_like_whole_string_match values
+ (1, 'acb'),
+ (2, concat('acb', char(10))),
+ (3, concat('acb', char(13), char(10))),
+ (4, concat('a', char(10), 'b')),
+ (5, 'acbx'),
+ (6, 'ab*'),
+ (7, concat('ab*', char(10))),
+ (8, 'ab*xyz');
+ """
+
+ // a value that ends with a newline is one character longer, so it must
not match a
+ // pattern that is anchored at the tail
+ qt_anchor_underscore """
+ select id, length(s) as len, s like 'a_b' as r from
test_like_whole_string_match order by id
+ """
+ qt_anchor_leading_percent """
+ select id, length(s) as len, s like '%c%b' as r from
test_like_whole_string_match order by id
+ """
+ // a literal '*' at the end of the pattern does not make it end with a
wildcard
+ qt_literal_star_tail """
+ select id, length(s) as len, s like 'a_*' as r from
test_like_whole_string_match order by id
+ """
+ // a pattern that really ends with '%' still accepts anything, the newline
included
+ qt_trailing_percent """
+ select id, length(s) as len, s like 'a_b%' as r from
test_like_whole_string_match order by id
+ """
+
+ // a pattern without wildcards is rewritten to an equality by
LIKE_TO_EQUAL, which would
+ // make this block pass with LIKE broken; disable the rule so the shortcut
path really runs
+ sql "set disable_nereids_expression_rules='LIKE_TO_EQUAL'"
+ qt_equals_shortcut """
+ select id, length(s) as len, s like 'acb' as r from
test_like_whole_string_match order by id
+ """
+
+ // a LIKE in a filter with function pushdown enabled is evaluated by
LikeColumnPredicate,
+ // which converts the pattern a second time through
LikeSearchState::clone()
+ sql "set enable_function_pushdown = true"
+ qt_pushdown_underscore """
+ select id from test_like_whole_string_match where s like 'a_b' order
by id
+ """
+ qt_pushdown_literal_star """
+ select id from test_like_whole_string_match where s like 'a_*' order
by id
+ """
+ // NOT LIKE reaches the scan as CompoundPredicate(NOT, LIKE), whose
children are not a slot,
+ // so it stays a generic expression rather than a LikeColumnPredicate
+ qt_not_literal_star """
+ select id from test_like_whole_string_match where s not like 'a_*'
order by id
+ """
+ qt_pushdown_trailing_percent """
+ select id from test_like_whole_string_match where s like 'a_b%' order
by id
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
