This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 7b457d617d6 branch-4.2: [feat](inverted-index) Support configurable
ngram size difference #67917 (#68774)
7b457d617d6 is described below
commit 7b457d617d6a772ad5fd7a21131028e5cabc7d28
Author: Jack <[email protected]>
AuthorDate: Fri Oct 16 07:34:40 2026 +0800
branch-4.2: [feat](inverted-index) Support configurable ngram size
difference #67917 (#68774)
### What problem does this PR solve?
Issue Number: close #67916
Related PR: #67917
Problem Summary:
Cherry-pick of #67917 (commit 6a79f954e26, a squash merge) to
branch-4.2. Custom ngram tokenizers can now set `max_ngram_diff` (0 to
255, default 1) instead of the hard-coded difference of 1. FE and BE
validate the same range, and newly created policies cap absolute gram
sizes at 1024. A persisted `max_ngram_diff` marker keeps marker-less
policies from before the change valid. `max_ngram_diff` does not change
the analyzer identity.
branch-4.2 also has the gram `mode` tokenizer, which the source change
does not know about. Adaptations:
- `max_ngram_diff` and the size limits apply to the legacy (no `mode`)
path only.
- `max_ngram_diff` together with `mode` is rejected.
- The compatibility marker is added only to new legacy ngram policies.
### Release note
Allow custom ngram tokenizers to configure the maximum difference
between max_gram and min_gram with max_ngram_diff values from 0 through
255.
### Check List (For Author)
- Test
- [ ] Regression test (the case is included; it was not run locally and
is left to CI)
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. New custom ngram tokenizers can opt into a wider gram-size
range (difference up to 255, absolute size up to 1024).
- Does this need documentation?
- [ ] No.
- [x] Yes. Same documentation as the source PR.
### Validation
- FE: `run-fe-ut.sh --run
PolicyValidatorTests,AnalyzerIdentityBuilderTest`: 117 tests, 0
failures; Checkstyle passed.
- BE: `NGramTokenizerTest.*`: 21/21 passed (includes the 7 new tests),
ASAN.
- clang-format 16 and Checkstyle merge-tree preflight: PASS.
- Local-environment notes (not part of this PR): `origin/branch-4.2`
currently declares thrift field id 229 twice in
`PaloInternalService.thrift` (`enable_paimon_rust_reader`,
`enable_prune_nested_column`), so a clean branch-4.2 build fails; I
bumped one id locally only to run the tests. For the BE UT I also
stubbed the unrelated paimon_rust sources locally and linked with
`--unresolved-symbols=ignore-all`. None of this is in the commit.
### Hunk audit (against #67917)
| File | Hunk | Result |
|---|---|---|
| ngram_tokenizer.cpp | `@@ NGramTokenizer::init` buffer size |
Ported(verbatim) |
| ngram_tokenizer.h | trailing newline | Ported (4.2 keeps
`compile_check_end.h`) |
| ngram_tokenizer_factory.cpp | `initialize` limits and diff check,
newline | Ported(verbatim); the 4.2 gram branch returns earlier, so
these apply to the legacy path only |
| ngram_tokenizer_factory.h | `MAX_NGRAM_DIFF`/`MAX_NGRAM_SIZE`
constants | Ported(verbatim) |
| ngram_tokenizer_test.cpp | 7 new tests | Ported; 4.2's
`RejectsMalformedUtf8AndCanBeReset` kept |
| NGramTokenizerValidator.java | `ALLOWED_PROPS`, constants,
`isValidPolicy`, size/diff checks | Adapted: 4.2 `mode` props kept,
`max_ngram_diff` rejected with `mode` |
| IndexPolicy.java | `@@ isInvalid` | Ported (uses `isValidPolicy`; same
behavior as 4.2's try/catch) |
| IndexPolicyMgr.java | `@@ validateReferencedComponentsUsableLocked` |
N-A: 4.2 already has it (tokenizer `isInvalid` check present) |
| IndexPolicyMgr.java | `@@ createIndexPolicy` marker | Adapted: only
for non-`mode` ngram |
| IndexPolicyMgr.java | `@@ validateReferencedPolicy` isInvalid check |
N-A: already present in 4.2 |
| IndexPolicyMgr.java | `@@ warnIfUnsupported` message | Ported |
| AnalyzerIdentityBuilder.java | `PROP_MAX_NGRAM_DIFF`, `isInvalid`
identity, `remove(max_ngram_diff)` | N-A: all three already present in
4.2 (verified with `git grep` on `origin/branch-4.2`) |
| AnalyzerIdentityBuilderTest.java | 3 tests | Ported; 4.2's
`testNgramDefaultBoundsDoNotChangeAnalyzerIdentity` kept |
| PolicyValidatorTests.java | difference/size/marker tests | Ported; 4.2
gram-mode tests kept; 1 new test for `max_ngram_diff` + `mode` |
| regression-test .groovy/.out | new files | Ported(verbatim) |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
---
.../inverted/tokenizer/ngram/ngram_tokenizer.cpp | 3 +-
.../inverted/tokenizer/ngram/ngram_tokenizer.h | 2 +-
.../tokenizer/ngram/ngram_tokenizer_factory.cpp | 29 ++++-
.../tokenizer/ngram/ngram_tokenizer_factory.h | 7 +-
.../inverted/tokenizer/ngram_tokenizer_test.cpp | 81 +++++++++++++
.../org/apache/doris/indexpolicy/IndexPolicy.java | 24 ++--
.../apache/doris/indexpolicy/IndexPolicyMgr.java | 7 ++
.../doris/indexpolicy/NGramTokenizerValidator.java | 58 ++++++++-
.../invertedindex/AnalyzerIdentityBuilderTest.java | 42 ++++++-
.../doris/indexpolicy/PolicyValidatorTests.java | 134 +++++++++++++++++++++
.../test_ngram_max_diff_custom_analyzer.out | 4 +
.../test_ngram_max_diff_custom_analyzer.groovy | 69 +++++++++++
12 files changed, 434 insertions(+), 26 deletions(-)
diff --git a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp
b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp
index e8fcbea9d79..d3d63b7a885 100644
--- a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp
+++ b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp
@@ -115,7 +115,8 @@ void NGramTokenizer::init(int32_t min_gram, int32_t
max_gram, bool edges_only) {
_min_gram = min_gram;
_max_gram = max_gram;
_edges_only = edges_only;
- _buffer.resize(4 * max_gram + 1024);
+ const size_t buffer_size = static_cast<size_t>(max_gram) * 4 + 1024;
+ _buffer.resize(buffer_size);
}
void NGramTokenizer::update_last_non_token_char() {
diff --git a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h
b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h
index 55515b2a273..72b6110ea86 100644
--- a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h
+++ b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h
@@ -77,4 +77,4 @@ private:
};
#include "common/compile_check_end.h"
-} // namespace doris::segment_v2::inverted_index
\ No newline at end of file
+} // namespace doris::segment_v2::inverted_index
diff --git
a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.cpp
b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.cpp
index ea39b2ac169..bb95a29301c 100644
--- a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.cpp
+++ b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.cpp
@@ -56,12 +56,35 @@ void NGramTokenizerFactory::initialize(const Settings&
settings) {
_min_gram = settings.get_int("min_gram",
NGramTokenizer::DEFAULT_MIN_NGRAM_SIZE);
_max_gram = settings.get_int("max_gram",
NGramTokenizer::DEFAULT_MAX_NGRAM_SIZE);
+ if (_min_gram <= 0 || _max_gram <= 0) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_gram and max_gram
must be positive");
+ }
+ if (_min_gram > _max_gram) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_gram must not be
greater than max_gram");
+ }
+ const bool has_max_ngram_diff =
!settings.get_string("max_ngram_diff").empty();
+ if (has_max_ngram_diff && (_min_gram > MAX_NGRAM_SIZE || _max_gram >
MAX_NGRAM_SIZE)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "min_gram and max_gram must be less than or equal to "
+
+ std::to_string(MAX_NGRAM_SIZE));
+ }
+ int32_t max_ngram_diff = settings.get_int("max_ngram_diff", 1);
+ if (max_ngram_diff < 0) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "max_ngram_diff must be greater than or equal to 0");
+ }
+ if (max_ngram_diff > MAX_NGRAM_DIFF) {
+ throw Exception(
+ ErrorCode::INVALID_ARGUMENT,
+ "max_ngram_diff must be less than or equal to " +
std::to_string(MAX_NGRAM_DIFF));
+ }
int32_t ngram_diff = _max_gram - _min_gram;
- if (ngram_diff > 1) {
+ if (ngram_diff > max_ngram_diff) {
throw Exception(
ErrorCode::INVALID_ARGUMENT,
"The difference between max_gram and min_gram in NGram
Tokenizer must be less "
- "than or equal to: [ 1 ] but was [" +
+ "than or equal to: [ " +
+ std::to_string(max_ngram_diff) + " ] but was [" +
std::to_string(ngram_diff) + "]");
}
_matcher = parse_token_chars(settings);
@@ -110,4 +133,4 @@ CharMatcherPtr
NGramTokenizerFactory::parse_token_chars(const Settings& settings
return builder.build();
}
-} // namespace doris::segment_v2::inverted_index
\ No newline at end of file
+} // namespace doris::segment_v2::inverted_index
diff --git
a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.h
b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.h
index 396ed67a1d3..464c5ff3db8 100644
--- a/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.h
+++ b/be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.h
@@ -31,6 +31,11 @@ namespace doris::segment_v2::inverted_index {
class NGramTokenizerFactory : public TokenizerFactory {
public:
+ // A configured range can emit one token per gram size at every input
position.
+ static constexpr int32_t MAX_NGRAM_DIFF = 255;
+ // Bound the per-stream buffer while retaining support for large
application-specific grams.
+ static constexpr int32_t MAX_NGRAM_SIZE = 1024;
+
NGramTokenizerFactory() = default;
~NGramTokenizerFactory() override = default;
@@ -81,4 +86,4 @@ private:
std::optional<gram::GramScheme> _gram_scheme; // set when "mode" is
present: create() goes gram
};
-}; // namespace doris::segment_v2::inverted_index
\ No newline at end of file
+}; // namespace doris::segment_v2::inverted_index
diff --git a/be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp
b/be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp
index 8529b6b081d..a82e8b8b56b 100644
--- a/be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp
+++ b/be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp
@@ -110,6 +110,87 @@ TEST(NGramTokenizerTest, InvalidMinMaxDifference) {
ASSERT_TRUE(exception_thrown);
}
+TEST(NGramTokenizerTest, ConfiguredMinMaxDifference) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["min_gram"] = "1";
+ args["max_gram"] = "8";
+ args["max_ngram_diff"] = "7";
+ Settings settings(args);
+ factory.initialize(settings);
+ auto tokens = tokenize(factory, "abcdefgh");
+
+ std::vector<std::string> expected {
+ "a", "ab", "abc", "abcd", "abcde", "abcdef",
"abcdefg", "abcdefgh", "b",
+ "bc", "bcd", "bcde", "bcdef", "bcdefg", "bcdefgh", "c",
"cd", "cde",
+ "cdef", "cdefg", "cdefgh", "d", "de", "def", "defg",
"defgh", "e",
+ "ef", "efg", "efgh", "f", "fg", "fgh", "g",
"gh", "h"};
+ ASSERT_EQ(tokens, expected);
+}
+
+TEST(NGramTokenizerTest, InvalidConfiguredDifferenceLimit) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["max_ngram_diff"] = "-1";
+ Settings settings(args);
+
+ EXPECT_THROW(factory.initialize(settings), Exception);
+}
+
+TEST(NGramTokenizerTest, ExcessiveConfiguredDifferenceLimit) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["max_ngram_diff"] =
std::to_string(NGramTokenizerFactory::MAX_NGRAM_DIFF + 1);
+ Settings settings(args);
+
+ EXPECT_THROW(factory.initialize(settings), Exception);
+}
+
+TEST(NGramTokenizerTest, ConfiguredDifferenceLimitBoundary) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["min_gram"] = "1";
+ args["max_gram"] = std::to_string(NGramTokenizerFactory::MAX_NGRAM_DIFF +
1);
+ args["max_ngram_diff"] =
std::to_string(NGramTokenizerFactory::MAX_NGRAM_DIFF);
+ Settings settings(args);
+
+ EXPECT_NO_THROW(factory.initialize(settings));
+}
+
+TEST(NGramTokenizerTest, AbsoluteSizeBoundary) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["min_gram"] = std::to_string(NGramTokenizerFactory::MAX_NGRAM_SIZE);
+ args["max_gram"] = std::to_string(NGramTokenizerFactory::MAX_NGRAM_SIZE);
+ args["max_ngram_diff"] = "1";
+ Settings settings(args);
+
+ EXPECT_NO_THROW(factory.initialize(settings));
+ EXPECT_NO_THROW(factory.create());
+}
+
+TEST(NGramTokenizerTest, ExcessiveAbsoluteSize) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["min_gram"] = std::to_string(NGramTokenizerFactory::MAX_NGRAM_SIZE);
+ args["max_gram"] = std::to_string(NGramTokenizerFactory::MAX_NGRAM_SIZE +
1);
+ args["max_ngram_diff"] = "1";
+ Settings settings(args);
+
+ EXPECT_THROW(factory.initialize(settings), Exception);
+}
+
+TEST(NGramTokenizerTest, LegacyFixedSizeAboveCurrentLimit) {
+ NGramTokenizerFactory factory;
+ std::unordered_map<std::string, std::string> args;
+ args["min_gram"] = "2048";
+ args["max_gram"] = "2048";
+ Settings settings(args);
+
+ EXPECT_NO_THROW(factory.initialize(settings));
+ EXPECT_NO_THROW(factory.create());
+}
+
TEST(NGramTokenizerTest, SymbolCharactersHandling) {
NGramTokenizerFactory factory;
std::unordered_map<std::string, std::string> args;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java
b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java
index 9075d51bf26..0fdee7848e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java
@@ -130,21 +130,13 @@ public class IndexPolicy implements Writable,
GsonPostProcessable {
ImmutableSet.of("common_grams");
public boolean isInvalid() {
- if (properties == null) {
- return false;
- }
- if (type == IndexPolicyTypeEnum.TOKEN_FILTER
- &&
LEGACY_UNSUPPORTED_TOKEN_FILTER_TYPES.contains(properties.get(PROP_TYPE))) {
- return true;
- }
- if (type == IndexPolicyTypeEnum.TOKENIZER
- && "ngram".equals(properties.get(PROP_TYPE))) {
- try {
- new NGramTokenizerValidator().validate(properties);
- } catch (DdlException | RuntimeException e) {
- return true;
- }
- }
- return false;
+ boolean hasUnsupportedTokenFilter = type ==
IndexPolicyTypeEnum.TOKEN_FILTER
+ && properties != null
+ &&
LEGACY_UNSUPPORTED_TOKEN_FILTER_TYPES.contains(properties.get(PROP_TYPE));
+ boolean hasInvalidNgramTokenizer = type ==
IndexPolicyTypeEnum.TOKENIZER
+ && properties != null
+ && "ngram".equals(properties.get(PROP_TYPE))
+ && !NGramTokenizerValidator.isValidPolicy(properties);
+ return hasUnsupportedTokenFilter || hasInvalidNgramTokenizer;
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
index beb758cac5b..e226d966206 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java
@@ -380,6 +380,13 @@ public class IndexPolicyMgr implements Writable,
GsonPostProcessable {
Map<String, String> storedProperties = properties == null
? null : Maps.newHashMap(properties);
validatePolicyProperties(type, storedProperties);
+ if (type == IndexPolicyTypeEnum.TOKENIZER
+ &&
"ngram".equals(storedProperties.get(IndexPolicy.PROP_TYPE))
+ && !storedProperties.containsKey("mode")) {
+ // Presence distinguishes policies created with the
absolute-size limit from
+ // compatible policies replayed from a version before
max_ngram_diff existed.
+ storedProperties.putIfAbsent("max_ngram_diff", "1");
+ }
IndexPolicy indexPolicy = IndexPolicy.create(policyName, type,
storedProperties);
if (nameToIndexPolicy.containsKey(normalizedName)) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/NGramTokenizerValidator.java
b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/NGramTokenizerValidator.java
index 2839ad568cf..869be8fc43c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/NGramTokenizerValidator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/NGramTokenizerValidator.java
@@ -28,8 +28,13 @@ import java.util.Set;
import java.util.regex.Pattern;
public class NGramTokenizerValidator extends BasePolicyValidator {
+ // A configured range can emit one token per gram size at every input
position.
+ static final int MAX_NGRAM_DIFF = 255;
+ // NGramTokenizer keeps four code-point slots per configured gram plus a
refill margin.
+ static final int MAX_NGRAM_SIZE = 1024;
+
private static final Set<String> ALLOWED_PROPS = ImmutableSet.of(
- "type", "min_gram", "max_gram", "token_chars",
"custom_token_chars",
+ "type", "min_gram", "max_gram", "max_ngram_diff", "token_chars",
"custom_token_chars",
"mode", "density", "lower_case");
private static final Set<String> VALID_TOKEN_CHARS = ImmutableSet.of(
@@ -47,8 +52,27 @@ public class NGramTokenizerValidator extends
BasePolicyValidator {
"[+-]?(?:[0-9]+(?:\\.[0-9]*)?|\\.[0-9]+)(?:[eE][+-]?[0-9]+)?");
private static final Pattern INTEGER_PATTERN =
Pattern.compile("[+-]?[0-9]+");
+ private final boolean enforceAbsoluteSizeLimit;
+
public NGramTokenizerValidator() {
+ this(true);
+ }
+
+ private NGramTokenizerValidator(boolean enforceAbsoluteSizeLimit) {
super(ALLOWED_PROPS);
+ this.enforceAbsoluteSizeLimit = enforceAbsoluteSizeLimit;
+ }
+
+ static boolean isValidPolicy(Map<String, String> properties) {
+ try {
+ // Policies created before max_ngram_diff existed have no
compatibility marker and
+ // must retain the absolute-size behavior accepted by the previous
release.
+ boolean hasCompatibilityMarker =
properties.containsKey("max_ngram_diff");
+ new
NGramTokenizerValidator(hasCompatibilityMarker).validate(properties);
+ return true;
+ } catch (DdlException | RuntimeException e) {
+ return false;
+ }
}
@Override
@@ -101,6 +125,35 @@ public class NGramTokenizerValidator extends
BasePolicyValidator {
throw new DdlException("max_gram [" + maxGram + "] "
+ "cannot be smaller than min_gram [" + minGram + "]");
}
+ if (enforceAbsoluteSizeLimit
+ && (minGram > MAX_NGRAM_SIZE || maxGram > MAX_NGRAM_SIZE)) {
+ throw new DdlException("min_gram and max_gram must be less than or
equal to " + MAX_NGRAM_SIZE);
+ }
+
+ int maxNgramDiff = 1;
+ if (props.containsKey("max_ngram_diff")) {
+ String value = props.get("max_ngram_diff");
+ if (!value.matches("-?[0-9]+")) {
+ throw new DdlException("max_ngram_diff must be a non-negative
integer");
+ }
+ try {
+ maxNgramDiff = Integer.parseInt(value);
+ if (maxNgramDiff < 0) {
+ throw new DdlException("max_ngram_diff must be greater
than or equal to 0");
+ }
+ if (maxNgramDiff > MAX_NGRAM_DIFF) {
+ throw new DdlException("max_ngram_diff must be less than
or equal to " + MAX_NGRAM_DIFF);
+ }
+ } catch (NumberFormatException e) {
+ throw new DdlException("max_ngram_diff must be a non-negative
integer");
+ }
+ }
+
+ int ngramDiff = maxGram - minGram;
+ if (ngramDiff > maxNgramDiff) {
+ throw new DdlException("The difference between max_gram and
min_gram in NGram Tokenizer must be less "
+ + "than or equal to: [ " + maxNgramDiff + " ] but was [" +
ngramDiff + "]");
+ }
if (props.containsKey("token_chars")) {
String tokenChars = props.get("token_chars");
@@ -134,6 +187,9 @@ public class NGramTokenizerValidator extends
BasePolicyValidator {
throw new DdlException("ngram tokenizer mode must be one of " +
VALID_MODES
+ ", got: '" + mode + "'" + (mode.isEmpty() ? " (empty)" :
""));
}
+ if (props.containsKey("max_ngram_diff")) {
+ throw new DdlException("max_ngram_diff cannot be used together
with mode");
+ }
int minGram = parseIntInRange(props, "min_gram", GRAM_MIN_GRAM_DEFAULT,
MIN_GRAM_LOWER_BOUND, MIN_GRAM_UPPER_BOUND);
int maxGram = parseIntInRange(props, "max_gram", GRAM_MAX_GRAM_DEFAULT,
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
index 818b834da5b..0425abf32d5 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java
@@ -150,6 +150,42 @@ public class AnalyzerIdentityBuilderTest {
}
}
+ @Test
+ public void testNgramValidationLimitDoesNotChangeAnalyzerIdentity() {
+ IndexPolicyMgr policyMgr = Mockito.mock(IndexPolicyMgr.class);
+ Env env = Mockito.mock(Env.class);
+ Mockito.when(env.getIndexPolicyMgr()).thenReturn(policyMgr);
+
+ Map<String, String> tokenizerProps = new HashMap<>();
+ tokenizerProps.put(IndexPolicy.PROP_TYPE, "ngram");
+ tokenizerProps.put("min_gram", "1");
+ tokenizerProps.put("max_gram", "2");
+ tokenizerProps.put("max_ngram_diff", "7");
+ IndexPolicy tokenizerWithLimit = new IndexPolicy(
+ 1, "ngram_with_limit", IndexPolicyTypeEnum.TOKENIZER,
tokenizerProps);
+
+ Map<String, String> equivalentTokenizerProps = new
HashMap<>(tokenizerProps);
+ equivalentTokenizerProps.remove("max_ngram_diff");
+ IndexPolicy tokenizerWithoutLimit = new IndexPolicy(
+ 2, "ngram_without_limit", IndexPolicyTypeEnum.TOKENIZER,
equivalentTokenizerProps);
+
+ IndexPolicy analyzerWithLimit = analyzerPolicy(3,
"analyzer_with_limit", "ngram_with_limit");
+ IndexPolicy analyzerWithoutLimit = analyzerPolicy(4,
"analyzer_without_limit", "ngram_without_limit");
+
Mockito.when(policyMgr.getPolicyByName("ngram_with_limit")).thenReturn(tokenizerWithLimit);
+
Mockito.when(policyMgr.getPolicyByName("ngram_without_limit")).thenReturn(tokenizerWithoutLimit);
+
Mockito.when(policyMgr.getPolicyByName("analyzer_with_limit")).thenReturn(analyzerWithLimit);
+
Mockito.when(policyMgr.getPolicyByName("analyzer_without_limit")).thenReturn(analyzerWithoutLimit);
+
+ try (MockedStatic<Env> mockedEnv = Mockito.mockStatic(Env.class)) {
+ mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
+ String identityWithLimit =
AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(), "analyzer_with_limit", "",
"__default__", "none", null);
+ String identityWithoutLimit =
AnalyzerIdentityBuilder.buildAnalyzerIdentity(
+ nonEmptyProperties(), "analyzer_without_limit", "",
"__default__", "none", null);
+ Assertions.assertEquals(identityWithoutLimit, identityWithLimit);
+ }
+ }
+
@Test
public void testReplayedInvalidNgramDoesNotBlockValidReplacement() throws
Exception {
IndexPolicyMgr policyMgr = new IndexPolicyMgr();
@@ -158,13 +194,13 @@ public class AnalyzerIdentityBuilderTest {
Map<String, String> invalidProps = new HashMap<>();
invalidProps.put(IndexPolicy.PROP_TYPE, "ngram");
- invalidProps.put("min_gram", "0");
- invalidProps.put("max_gram", "2");
+ invalidProps.put("min_gram", "1");
+ invalidProps.put("max_gram", "8");
IndexPolicy invalidTokenizer = new IndexPolicy(
10, "replayed_ngram", IndexPolicyTypeEnum.TOKENIZER,
invalidProps);
Map<String, String> replacementProps = new HashMap<>(invalidProps);
- replacementProps.put("min_gram", "1");
+ replacementProps.put("max_ngram_diff", "7");
IndexPolicy replacementTokenizer = new IndexPolicy(
11, "replacement_ngram", IndexPolicyTypeEnum.TOKENIZER,
replacementProps);
IndexPolicy invalidAnalyzer = analyzerPolicy(12, "replayed_analyzer",
"replayed_ngram");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java
b/fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java
index 8dfb182a881..24c16d181f4 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java
@@ -141,6 +141,7 @@ public class PolicyValidatorTests {
Map<String, String> props = new HashMap<>();
props.put("min_gram", "3");
props.put("max_gram", "5");
+ props.put("max_ngram_diff", "2");
validator.validate(props); // Should not throw
}
@@ -157,6 +158,15 @@ public class PolicyValidatorTests {
validator.validate(props);
}
+ @Test
+ public void testNGramValidator_GramModeRejectsMaxNgramDiff() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = sparseGramProps();
+ props.put("max_ngram_diff", "7");
+ DdlException e = Assertions.assertThrows(DdlException.class, () ->
validator.validate(props));
+ Assertions.assertTrue(e.getMessage().contains("max_ngram_diff cannot
be used together with mode"));
+ }
+
@Test
public void testNGramValidator_GramModeRejectsBadValues() {
NGramTokenizerValidator validator = new NGramTokenizerValidator();
@@ -331,6 +341,130 @@ public class PolicyValidatorTests {
Assertions.assertFalse(replayed.isInvalid());
}
+ @Test
+ public void testNGramValidator_DefaultDifferenceLimit() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("min_gram", "1");
+ props.put("max_gram", "8");
+
+ Exception exception = Assertions.assertThrows(DdlException.class,
+ () -> validator.validate(props));
+ Assertions.assertTrue(exception.getMessage().contains("less than or
equal to: [ 1 ]"));
+ }
+
+ @Test
+ public void testNGramValidator_ConfiguredDifferenceLimit() throws
Exception {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("min_gram", "1");
+ props.put("max_gram", "8");
+ props.put("max_ngram_diff", "7");
+ validator.validate(props); // Should not throw
+ }
+
+ @Test
+ public void testNGramValidator_InvalidDifferenceLimit() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("max_ngram_diff", "-1");
+
+ Exception exception = Assertions.assertThrows(DdlException.class,
+ () -> validator.validate(props));
+ Assertions.assertTrue(exception.getMessage().contains("greater than or
equal to 0"));
+ }
+
+ @Test
+ public void testNGramValidator_RejectsNonAsciiDifferenceLimit() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("max_ngram_diff", "Ù§");
+
+ Exception exception = Assertions.assertThrows(DdlException.class,
+ () -> validator.validate(props));
+ Assertions.assertTrue(exception.getMessage().contains("non-negative
integer"));
+ }
+
+ @Test
+ public void testNGramValidator_RejectsExcessiveDifferenceLimit() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("max_ngram_diff",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_DIFF + 1));
+
+ Exception exception = Assertions.assertThrows(DdlException.class,
+ () -> validator.validate(props));
+ Assertions.assertTrue(exception.getMessage().contains("less than or
equal to 255"));
+ }
+
+ @Test
+ public void testNGramValidator_AcceptsDifferenceLimitBoundary() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("min_gram", "1");
+ props.put("max_gram",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_DIFF + 1));
+ props.put("max_ngram_diff",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_DIFF));
+
+ Assertions.assertDoesNotThrow(() -> validator.validate(props));
+ }
+
+ @Test
+ public void testNGramValidator_AcceptsAbsoluteSizeBoundary() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("min_gram",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_SIZE));
+ props.put("max_gram",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_SIZE));
+
+ Assertions.assertDoesNotThrow(() -> validator.validate(props));
+ }
+
+ @Test
+ public void testNGramValidator_RejectsExcessiveAbsoluteSize() {
+ NGramTokenizerValidator validator = new NGramTokenizerValidator();
+ Map<String, String> props = new HashMap<>();
+ props.put("min_gram",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_SIZE));
+ props.put("max_gram",
Integer.toString(NGramTokenizerValidator.MAX_NGRAM_SIZE + 1));
+
+ Exception exception = Assertions.assertThrows(DdlException.class,
+ () -> validator.validate(props));
+ Assertions.assertTrue(exception.getMessage().contains("less than or
equal to 1024"));
+ }
+
+ @Test
+ public void
testLegacyNGramPolicyAboveCurrentLimitRemainsValidAfterReplay() throws
Exception {
+ Map<String, String> props = new HashMap<>();
+ props.put(IndexPolicy.PROP_TYPE, "ngram");
+ props.put("min_gram", "2048");
+ props.put("max_gram", "2048");
+
+ IndexPolicy replayed = roundTrip(new IndexPolicy(
+ 1, "legacy_large_ngram", IndexPolicyTypeEnum.TOKENIZER,
props));
+
+ Assertions.assertFalse(replayed.isInvalid());
+
+ props.put("max_ngram_diff", "1");
+ IndexPolicy current = roundTrip(new IndexPolicy(
+ 2, "current_large_ngram", IndexPolicyTypeEnum.TOKENIZER,
props));
+ Assertions.assertTrue(current.isInvalid());
+ }
+
+ @Test
+ public void testNewNGramPolicyPersistsCompatibilityMarker() throws
Exception {
+ Env env = Mockito.mock(Env.class);
+ Mockito.when(env.getNextId()).thenReturn(2L);
+ Mockito.when(env.getEditLog()).thenReturn(Mockito.mock(EditLog.class));
+ IndexPolicyMgr policyMgr = new IndexPolicyMgr();
+ Map<String, String> props = new HashMap<>();
+ props.put(IndexPolicy.PROP_TYPE, "ngram");
+
+ try (MockedStatic<Env> mockedEnv = Mockito.mockStatic(Env.class)) {
+ mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
+ policyMgr.createIndexPolicy(false, "new_ngram",
IndexPolicyTypeEnum.TOKENIZER, props);
+ }
+
+ Assertions.assertEquals("1",
+
policyMgr.getPolicyByName("new_ngram").getProperties().get("max_ngram_diff"));
+ }
+
// StandardTokenizerValidator Tests
@Test
public void testStandardTokenizerValidator_ValidProperties() throws
Exception {
diff --git
a/regression-test/data/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.out
b/regression-test/data/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.out
new file mode 100644
index 00000000000..07b08b46c6e
--- /dev/null
+++
b/regression-test/data/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !ngram_tokens --
+[{\n "token": "a"\n }, {\n "token": "ab"\n }, {\n
"token": "abc"\n }, {\n "token": "abcd"\n }, {\n "token":
"abcde"\n }, {\n "token": "abcdef"\n }, {\n "token":
"abcdefg"\n }, {\n "token": "abcdefgh"\n }, {\n "token":
"b"\n }, {\n "token": "bc"\n }, {\n "token": "bcd"\n },
{\n "token": "bcde"\n }, {\n "token": "bcdef"\n }, {\n
"token": "bcdefg"\n }, [...]
+
diff --git
a/regression-test/suites/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.groovy
b/regression-test/suites/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.groovy
new file mode 100644
index 00000000000..5e9a84e86d9
--- /dev/null
+++
b/regression-test/suites/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.groovy
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_ngram_max_diff_custom_analyzer", "p0") {
+ def defaultLimitTokenizer = "test_ngram_default_limit_tokenizer"
+ def ngramTokenizer = "test_ngram_1_8_tokenizer"
+ def ngramAnalyzer = "test_ngram_1_8_analyzer"
+
+ try_sql "DROP INVERTED INDEX ANALYZER IF EXISTS ${ngramAnalyzer}"
+ try_sql "DROP INVERTED INDEX TOKENIZER IF EXISTS ${defaultLimitTokenizer}"
+ try_sql "DROP INVERTED INDEX TOKENIZER IF EXISTS ${ngramTokenizer}"
+
+ test {
+ sql """
+ CREATE INVERTED INDEX TOKENIZER ${defaultLimitTokenizer}
+ PROPERTIES (
+ "type" = "ngram",
+ "min_gram" = "1",
+ "max_gram" = "8"
+ )
+ """
+ exception "less than or equal to: [ 1 ]"
+ }
+
+ sql """
+ CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ${ngramTokenizer}
+ PROPERTIES (
+ "type" = "ngram",
+ "min_gram" = "1",
+ "max_gram" = "8",
+ "max_ngram_diff" = "7"
+ )
+ """
+ sql """
+ CREATE INVERTED INDEX ANALYZER IF NOT EXISTS ${ngramAnalyzer}
+ PROPERTIES ("tokenizer" = "${ngramTokenizer}")
+ """
+
+ int maxRetry = 30
+ Exception lastException = null
+ for (int i = 0; i < maxRetry; i++) {
+ try {
+ sql """SELECT TOKENIZE('probe', '"analyzer"="${ngramAnalyzer}"')"""
+ lastException = null
+ break
+ } catch (Exception e) {
+ lastException = e
+ sleep(1000)
+ }
+ }
+ assertTrue(lastException == null,
+ "Analyzer ${ngramAnalyzer} was not ready:
${lastException?.message}")
+
+ qt_ngram_tokens """SELECT TOKENIZE('abcdefgh',
'"analyzer"="${ngramAnalyzer}"')"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
