Gabriel39 commented on code in PR #69672:
URL: https://github.com/apache/doris/pull/69672#discussion_r4396777084
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java:
##########
@@ -251,26 +254,54 @@ private void serializeFieldContexts(EsMetadataState
state, Map<String, String> n
}
private EsQueryDslResult buildQueryDsl(Optional<ConnectorExpression>
filter,
- EsMetadataState state) {
+ EsMetadataState state, int docValuesMode) {
if (!filter.isPresent()) {
return new EsQueryDslResult("{\"match_all\":{}}",
Collections.emptyList());
}
Map<String, String> fieldsContext = Collections.emptyMap();
List<String> needCompatDateFields = Collections.emptyList();
Map<String, String> column2typeMap = Collections.emptyMap();
+ Set<String> unsafeFields = new HashSet<>();
if (state.getFieldContext() != null) {
if (props.isKeywordSniff()) {
fieldsContext =
state.getFieldContext().getFetchFieldsContext();
}
needCompatDateFields =
state.getFieldContext().getNeedCompatDateFields();
column2typeMap = state.getFieldContext().getColumn2typeMap();
+
unsafeFields.addAll(state.getFieldContext().getSourceValueUnsafeFields());
+ if (!props.isKeywordSniff()) {
+ // Without keyword sniff a term targets analyzed text, even
when a safe raw subfield
+ // exists. Do not apply the unused raw path's safety to the
actual text query.
+
unsafeFields.addAll(state.getFieldContext().getFetchFieldsContext().keySet());
+ }
+ }
+
+ if (docValuesMode == 0) {
+ collectSourceStringFields(filter.get(), unsafeFields);
}
return EsQueryDslBuilder.buildQueryDslWithResult(
filter.get(), fieldsContext, column2typeMap,
- props.isLikePushDown(), needCompatDateFields);
+ props.isLikePushDown(), needCompatDateFields, unsafeFields);
+ }
+
+ private static void collectSourceStringFields(ConnectorExpression
expression, Set<String> fields) {
+ if (expression instanceof ConnectorColumnRef) {
+ ConnectorColumnRef column = (ConnectorColumnRef) expression;
+ String type = column.getType().getTypeName();
+ // Source arrays are serialized as STRING, whereas ES predicates
match their elements.
+ // _id comes from hit metadata and cannot acquire an array
representation in source mode.
+ if (!"_id".equals(column.getColumnName())
+ && ("STRING".equalsIgnoreCase(type) ||
"VARCHAR".equalsIgnoreCase(type)
+ || "CHAR".equalsIgnoreCase(type))) {
Review Comment:
Fixed in 0a792c9461252986de54efcf38bd0d7d845c0612.
Fixed the normalizer-triggered regression and the equivalent `half_float`
case.
Mapping analysis now records rounded numeric fields in a source-mode-only
restriction set, unioned across all alias backing mappings and including
filter-only columns. Once the final read mode is known, source scans retain the
SQL residual and omit the narrowing numeric predicate. This does not remove
numeric docvalue eligibility: `SELECT b ... WHERE b=2.25` still uses docvalues
and returns 2.25.
I reproduced the previous head returning `(ok, 2.34)` for `SELECT a,b ...
WHERE b=2.25` in all four scanner/parallel-scroll combinations, alongside the
passing numeric-only control. With the fix, that source query is empty and
`b=2.34` retains the original row. The new standalone numeric regression covers
both rounded mappings, equality/range/IN/null/Boolean conditions, filter-only
columns, catalog and field-limit fallbacks, native `esquery` boundaries, and
pure numeric docvalue controls across 12 configurations.
Validation: 149 FE tests passed (one optional ES 9 compatibility class
skipped), FE Checkstyle passed, all 612 existing source-semantics checks
passed, and the numeric suite passed 264 stored checks plus 264 independent SQL
comparisons and 12 expected planning errors. Generated output was verified in a
separate run without output generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
