airborne12 opened a new pull request, #69419:
URL: https://github.com/apache/doris/pull/69419
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69292 (where review found it)
Problem Summary:
Predicate inference copies an OR on one column to every column that is equal
to it. When the OR contains MATCH, the copy changes the result:
```sql
CREATE TABLE t (id INT, c1 TEXT, c2 TEXT,
INDEX i1 (c1) USING INVERTED PROPERTIES ("parser" = "english"),
INDEX i2 (c2) USING INVERTED)
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
("replication_num" = "1");
INSERT INTO t VALUES (1, 'apple red', 'apple red'), (2, 'banana', 'banana');
SELECT id FROM t WHERE c1 = c2 AND (c1 MATCH_ANY 'apple' OR c1 MATCH_ANY
'pear');
-- no rows; row 1 is expected
```
`InferPredicateByReplace` added `(c2 MATCH_ANY 'apple' OR c2 MATCH_ANY
'pear')` to the scan. A MATCH depends on the inverted index of its column: `c2`
has no parser, so a MATCH on `c2` only matches a whole value, and the copy
dropped row 1.
When both columns have the same index, the copy is scored too. With `'apple
red'` and `'apple green'` in both columns, `score()` of the same filter with
`ORDER BY score() DESC LIMIT 10` doubles from 0.4700036 to 0.9400072. Adding
`score() > 0.6` returns both rows, although their score is below the minimum.
`InferPredicateByReplace` no longer infers from an OR that contains MATCH or
SEARCH; it never inferred from a MATCH outside an OR. SEARCH is left out for
the same reasons.
### Release note
Fix wrong results and scores when a query has an equality between two
columns and an OR of MATCH or SEARCH predicates on one of them.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Unit test:
`InferPredicateByReplaceTest.testDoNotInferMatchOrSearchInsideOr` checks that
an OR of MATCH and an OR with SEARCH next to `a = b` infer nothing. It fails
without this change, where the MATCH OR is copied to `b`.
`InferPredicateByReplaceTest`, `InferPredicatesTest` and
`UnequalPredicateInferTest` pass (104 tests). Checkstyle passes.
- Regression test:
`nereids_rules_p0/infer_predicate/infer_match_predicate` runs the queries
above. Its `.out` was generated by the framework on an FE built from this
change on master d30058a803, with a BE built from 41a58afba2 plus the
`SegmentIterator` change of #69292 (this change is FE only). On an FE built
from master, the same queries return no rows, 0.9400072 and two rows.
- The branch is based on 88511083a6, three commits before d30058a803,
because the fork cannot receive the workflow file change of #69386; those three
commits do not touch the changed files.
- Behavior changed:
- [ ] No.
- [x] Yes. Predicate inference no longer copies an OR that contains
MATCH or SEARCH to an equal column.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
