Gabriel39 opened a new pull request, #69650:
URL: https://github.com/apache/doris/pull/69650
### What problem does this PR solve?
Related PR: #69614
`external_table_p2/es/test_es_file_scanner_v2.groovy` fails on its first
legacy-scanner baseline query with `Table [test1] does not exist in database
[default_db]`. Its interpolated port key is a Groovy `GString`, while
`otherConfigs` is a `Properties` containing `String` keys.
`Properties.get(Object)` returns null even when the port is configured,
producing `http://<es-host>:null`. Convert the key to `String` and fail
explicitly when the configuration is absent. The failure happens before any V2
result comparison or BE scan.
The coverage review also found a query correctness issue shared by both
scanners: a nested AND can return a partial ES DSL and record an unsupported
child, but the single-expression path ignored that residual tracking. For `(a =
1 AND local_predicate) OR a = 2`, the engine could drop the local predicate;
`NOT(a = 1 AND local_predicate)` could also become an unsafe `NOT(a = 1)` at
the source. Retain the complete expression locally whenever conversion records
a residual. Fully pushable expressions and independent top-level AND conjuncts
retain their existing pushdown behavior.
Expand coverage in the same change:
- ES 5/6/7/8 V1/V2 comparisons across source/docvalue scans,
parallel/whole-index scrolling, and batch sizes 1/1024. Add
scalar/date/array/JSON, IN/NOT IN, null, raw ES query, residual filtering,
aggregation and LIMIT cases; restore every changed session variable.
- An independent equivalent-query oracle for nested residual OR, since a
V1/V2 comparison alone cannot detect a shared FE error. Compare cardinality for
unordered LIMIT rather than document identity across BEs.
- FE tests for residual tracking, docvalue eligibility and field-count
boundaries, alias/context serialization, shard planning and safe early-stop
parameters.
- BE tests for source-only fallback, missing/null docvalues and empty
arrays, real HTTP empty results, early close and scroll 404/500 cleanup. Add an
opt-in authenticated live-reader smoke test.
### Release note
Preserve local filtering for Elasticsearch compound predicates that cannot
be completely pushed down.
### Check List (For Author)
- Test
- [x] FE: all 127 ES connector tests pass. The two new nested OR/NOT
tests fail before the residual-tracking fix.
- [x] BE: all 27 selected ES ASAN tests pass against authenticated
Elasticsearch 9.0.0 and 9.5.5, including source/docvalue reads with batch size
1.
- [x] FE Checkstyle for the ES module and its reactor dependencies;
clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks.
- [x] Groovy 4.0.19: reproduce the null port before the fix; verify all
four port lookups, suite setup/matrix execution, missing configuration,
disabled execution and restoration after an injected query failure. The setup
probe substitutes SQL execution and does not validate SQL results.
- [ ] Full external ES 5/6/7/8 SQL regression run; requires the external
Docker fixtures and a Doris cluster.
- Behavior changed:
- [x] Yes: residual ES predicates remain enforced; regression setup
reads configured ports correctly.
- Does this need documentation?
- [x] No: existing catalog properties and scanner session variables are
unchanged.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
