github-actions[bot] commented on code in PR #69672:
URL: https://github.com/apache/doris/pull/69672#discussion_r4386047138
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsQueryDslBuilder.java:
##########
@@ -143,6 +158,11 @@ public static EsQueryDslResult
buildQueryDslWithResult(ConnectorExpression filte
return new EsQueryDslResult(queryDsl, notPushedIndices);
}
+ // Dropping a source-value predicate after translating a
transformed/missing index value
+ // cannot be repaired by a residual: matching source documents may
already be absent.
+ if (referencesUnsafeSourceField(filter, sourceValueUnsafeFields)) {
Review Comment:
[P2] Keep native esquery out of local Boolean residuals
With `a` mapped as `keyword` with `ignore_above=4`, a valid predicate such
as `a = 'cat' OR esquery(b, '{"term":{"b":"ok"}}')` becomes
`Scan[OR(EQ(a,'cat'), esquery(b,...))]`. The new unsafe-field check finds `a`
anywhere in that OR and returns remote `match_all` plus residual index 0.
`PluginDrivenScanNode.pruneConjunctsFromNodeProperties` then retains the entire
OR for BE evaluation, where `FunctionEsquery` is a fake function that returns
`esquery only supported on es table`. Previously both branches were translated
remotely; the short literal also avoids any preexisting ignore_above
discrepancy. Exempting the esquery node itself does not protect its containing
Boolean expression. Handle remote-only calls when choosing a residual and add
mixed esquery/unsafe-field OR and NOT coverage, so these queries do not reach
an unevaluable local function.
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsMappingUtils.java:
##########
@@ -153,9 +153,73 @@ public static EsFieldContext
resolveFieldContext(List<String> columnNames,
resolveKeywordFields(fieldContext, fieldObject, colName,
fieldType);
resolveDocValuesFields(fieldContext, fieldObject, colName,
fieldType);
}
+ resolveSourceValueSafety(fieldContext, indexMapping, mappingType);
return fieldContext;
}
+ private static void resolveSourceValueSafety(EsFieldContext context,
String indexMapping, String mappingType) {
+ try {
+ // An alias can span mappings. Every backing index must preserve
source semantics before
+ // a predicate may remove rows or doc_values may replace the value
returned to Doris.
+ for (JsonNode index : MAPPER.readTree(indexMapping)) {
+ ObjectNode root = getRootSchema((ObjectNode)
index.get("mappings"), mappingType, new ArrayList<>());
Review Comment:
[P2] Allow empty backing mappings during alias safety analysis
An alias can return a mapping response such as
`{"populated":{"mappings":{"properties":{"a":{"type":"long"}}}},"empty":{"mappings":{}}}`.
Schema inference and the original field resolution take the first populated
mapping, so this alias was readable with parallel scroll disabled even though
another backing index had no documents/mapping. The new loop calls
`getRootSchema` on the empty backing, where `checkNonPropertiesFields` throws
`Do not support index without explicit mapping.`;
`EsMetadataFetcher.fetchMapping` therefore aborts every scan before shard
planning. Treat an empty backing as an empty property set for safety analysis
(marking required fields unsafe as needed), rather than requiring every backing
to provide an explicit schema. Add a populated-first/empty-second alias test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
