github-actions[bot] commented on code in PR #69199:
URL: https://github.com/apache/doris/pull/69199#discussion_r4295209408
##########
be/src/storage/index/index_reader_helper.h:
##########
@@ -17,13 +17,26 @@
#pragma once
+#include <map>
+#include <string>
+
#include "storage/index/index_iterator.h"
#include "storage/index/inverted/inverted_index_reader.h"
namespace doris::segment_v2 {
class IndexReaderHelper {
public:
+ static bool is_normalizer_only(const std::map<std::string, std::string>&
properties) {
+ const auto normalizer =
properties.find(INVERTED_INDEX_NORMALIZER_NAME_KEY);
+ const auto analyzer =
properties.find(INVERTED_INDEX_ANALYZER_NAME_KEY);
+ // A normalizer keeps the whole value as one term, so phrase queries
need no positions.
Review Comment:
[P2] Retain positions for token-expanding custom normalizers
A `normalizer` property does not guarantee one output term: `CREATE INVERTED
INDEX NORMALIZER split_norm PROPERTIES("token_filter"="word_delimiter")` passes
FE validation and turns `foo-bar` into `foo` and `bar`. With the new FE
default, an SNII index using this normalizer stores no positions, but this
predicate admits `v MATCH_PHRASE 'foo-bar'`. Lowering retains a two-slot
phrase, `SniiIndexSource::open_terms(..., positions=true, ...)` returns
`NotSupported`, and `SegmentIterator` falls back to row evaluation. Previously
this query used the positional index; now selective phrase queries scan the
data. Preserve positions for normalizers that can expand tokens, or enforce a
single-token invariant before choosing docs-only storage, and cover this with a
real postings test. This differs from the existing lowercase-normalizer thread:
its singleton gate is fixed, while this permitted multi-token case needs
physical positions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
