924060929 commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4639794885
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -109,8 +121,120 @@ public List<ConnectorScanRange> planScan(ConnectorSession
session, ConnectorScan
// Include complete source roots so pruning cannot turn malformed
nested paths into missing values.
d.setProjectionBson(encode(projection));
}
- // One cursor owns the entire logical namespace. Filters and LIMIT
remain Doris residuals.
- return List.of(new MongoScanRange(d));
+ long descriptorSize = descriptorBytes(session, handle, d);
+ MongoSplitPlan plan = rangePlanner.plan(session, handle, snapshot,
d.getReadPreference(),
+ request.getScanContext(), descriptorSize);
+ // Retain the shared descriptor once; streaming creates owned RPC
values under engine backpressure.
+ // A wide schema must not be copied 256 times merely to keep a frozen
boundary table in FE.
+ return new AbstractList<>() {
+ @Override
+ public ConnectorScanRange get(int index) {
+ java.util.Objects.checkIndex(index, plan.size());
+ MongoRangePlanner.checkCancelled(session,
request.getScanContext());
+ TMongoScanDesc range = d.deepCopy();
+ TMongoSplit split = new TMongoSplit();
+ split.setStrategy(plan.family().isEmpty() ? "single" :
"range");
+ split.setField(plan.field());
+ split.setTypeFamily(plan.family());
+ split.setIndex(index);
+ split.setRequested(plan.requested());
+ split.setAdmitted(plan.admitted());
+ split.setActual(plan.size());
+ split.setReason(plan.reason());
+ if (!plan.family().isEmpty()) {
+ range.setVersion(2);
+ split.setFilterBson(plan.partitionBytes(index));
+ }
+ range.setSplit(split);
+ if (snapshot != null) {
+ range.setTimeoutMs(MongoSnapshot.remainingMillis(session));
+ }
+ if (plan.descriptorBudget() > 0 && descriptorSize >
plan.descriptorBudget()) {
Review Comment:
Is this guard reachable? Scan plan providers are created per scan node
(`PluginDrivenScanNode.resolveScanProvider()` memoizes one per node, and
`MongoConnector.getScanPlanProvider()` news one per call), and
`MongoRangePlanner.memoKey` embeds a per-provider UUID, so within one provider
`planScan` runs once and `descriptorSize == plan.descriptorBudget()` always
holds.
If a second `planScan` on the same provider really can supply a larger
candidate/projection, then any growth beyond roughly 8KB (a large `IN` list, a
60K-character literal - the same shape as the earlier review thread) makes a
legitimate query fail with `MongoDB descriptor grew after split plan freeze`,
where re-admitting the bytes or falling back to a `single` plan would be the
correct outcome. Please point at the path that makes this reachable, or drop
the guard.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoScanPlanProvider.java:
##########
@@ -132,19 +256,47 @@ public Map<String, String>
getScanNodeProperties(ConnectorSession session, Conne
Map<String, String> result = new LinkedHashMap<>();
var policies = session.getSessionProperties();
String concern =
properties.resolveReadConcern(policies.getOrDefault("mongodb_read_concern",
"inherit"));
- String preference = properties.resolveReadPreference(
- policies.getOrDefault("mongodb_read_preference", "inherit"));
- var snapshot = MongoSnapshot.prepare(session, properties, table,
client, concern, preference,
- 1000L * bounded(policies, "query_timeout", 300, 1,
Integer.MAX_VALUE / 1000));
+ var pushed = predicate(session, table, filter);
+ var pruned = projection(session, table, columns);
result.put("read_concern", concern);
- result.put("snapshot", snapshot == null ? "none" :
snapshot.getTimestampSeconds()
- + ":" + snapshot.getTimestampIncrement());
result.put("object_kind", table.getObjectKind());
result.put("schema_source", table.getSchema().getSource());
result.put("schema_version", table.getSchema().getVersion());
- result.put("predicate_pushdown", predicate(session, table,
filter).kind().name());
- result.put("projection_pushdown", projection(session, table,
columns).isEmpty() ? "false" : "true");
- return result;
+ result.put("predicate_pushdown", pushed.kind().name());
+ result.put("projection_pushdown", pruned.isEmpty() ? "false" : "true");
+ // initSchemaParams reads properties BEFORE WHERE/projection are
attached. Source probes
+ // and byte admission here would reject a normal large predicate and
ignore KILL QUERY.
+ // Resolve diagnostics lazily from the final cancellable plan, without
doing any source I/O.
+ return new AbstractMap<>() {
Review Comment:
Two issues with this lazily evaluated map:
1. By code reading, `EXPLAIN` goes `getNodeExplainString ->
getOrLoadScanNodeProperties -> appendExplainInfo`, none of which calls
`planScan`. So `rangePlanner.planned(...)` is empty at that point and this
branch yields the session strategy plus `planning_pending` / `0/0/0`. But
`test_mongodb_splits.groovy` asserts `requested/admitted/actual: 8/8/1` and
`reason: insufficient_boundaries` from `EXPLAIN`. Please confirm which holds -
if EXPLAIN really shows `planning_pending`, those assertions are currently
failing and the profile (evaluated after planning) will disagree with EXPLAIN
for the same query.
2. `predicate_pushdown` / `projection_pushdown` above are still computed
eagerly from the filter/columns that this very comment says are not attached
yet - same root cause, same cheap fix.
Minor: `AbstractMap` recomputes `entrySet()` on every access (`get()` is
O(n)) and two calls can observe different plan states. Returning a fresh
snapshot per call would be simpler and deterministic.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ static final int SAMPLE_ROWS = 4096;
+ static final int PROBE_TIMEOUT_MS = 2000;
Review Comment:
`PROBE_TIMEOUT_MS` is hard-coded and not session-tunable, while the probe
does `listIndexes` + `collStats` + `$sample(4096)` against the source. A probe
that misses this budget makes `range` fail with `MongoDB range planning
unavailable: probe_budget` (strict is defensible) and silently downgrades
`auto` to a single cursor. `SessionVariableMongoSplitTest` deliberately asserts
that `mongodb_split_plan_timeout_ms` is not settable, so please either document
why 2s is the right fixed budget on very large collections or expose a knob -
as written an operator cannot raise it short of `query_timeout`.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ static final int SAMPLE_ROWS = 4096;
+ static final int PROBE_TIMEOUT_MS = 2000;
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+ this.client = client;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ checkCancelled(session, context);
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ if ("single".equals(strategy)) {
+ return MongoSplitPlan.single(1, 1, "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
+ if (!custom) {
+ field = "_id";
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
field.length() > 256) {
+ throw new IllegalArgumentException("MongoDB split field must be an
unambiguous top-level field");
+ }
+ String selectedField = field;
+ Budget budget =
session.getStatementScope().computeIfAbsent("mongodb.split.budget",
Budget::new);
+ String tableKey = tableKey(session, table);
+ // Count fallback scans too: a view must not make a global custom
field look unambiguous.
+ if (custom) {
+ budget.customField(tableKey);
+ }
+ Memo memo = session.getStatementScope().computeIfAbsent(tableKey,
Memo::new);
+ return memo.get(session, context, () -> load(session, table, snapshot,
preference, context,
+ strategy, selectedField, descriptorBytes, budget));
+ }
+
+ Optional<MongoSplitPlan> planned(ConnectorSession session,
MongoTableHandle table) {
+ Memo memo = session.getStatementScope().getIfPresent(tableKey(session,
table));
+ return memo == null ? Optional.empty() : memo.completed();
+ }
+
+ private String tableKey(ConnectorSession session, MongoTableHandle table) {
+ // One provider can see several namespaces; never reuse another
collection's bounds/admission.
+ return memoKey + ":" + session.getCatalogId() + ":" +
table.getSchema().getVersion();
+ }
+
+ private MongoSplitPlan load(ConnectorSession session, MongoTableHandle
table, TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, String strategy,
String field,
+ long descriptorBytes, Budget budget) {
+ int maximum = splitLimit(session);
+ targetBytes(session, -1);
Review Comment:
The return value is discarded; the call's real purpose is to validate
`max_file_split_size` / `max_initial_file_split_size`. Please make that
explicit (`validateSplitSizes(session)`) - as written it reads like a forgotten
assignment.
The `(8L * 1024 * 1024) / descriptorBytes` on the next line only avoids a
division by zero because `descriptorBytes` is 0 exactly on the `single`
early-return path. A `> 0` assertion or a named constant would keep that
invariant local to this method instead of depending on a caller three frames
away.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoSplitPlan.java:
##########
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.bson.BsonArray;
+import org.bson.BsonDocument;
+import org.bson.BsonString;
+import org.bson.BsonValue;
+import org.bson.RawBsonDocument;
+import org.bson.codecs.BsonDocumentCodec;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+
+/** Frozen value partitions. Samples affect balance only; the unbounded ends
and complement ensure coverage. */
+final class MongoSplitPlan {
+ private final String field;
+ private final String family;
+ private final int requested;
+ private final int admitted;
+ private final String reason;
+ private final List<byte[]> partitions;
+ private final long descriptorBudget;
+
+ private MongoSplitPlan(String field, String family, int requested, int
admitted, String reason,
+ List<BsonDocument> partitions) {
+ this.field = field;
+ this.family = family;
+ this.requested = requested;
+ this.admitted = admitted;
+ this.reason = reason;
+ this.partitions =
partitions.stream().map(MongoSplitPlan::encode).toList();
+ this.descriptorBudget = 0;
+ }
+
+ private MongoSplitPlan(MongoSplitPlan plan, long descriptorBudget) {
+ this.field = plan.field;
+ this.family = plan.family;
+ this.requested = plan.requested;
+ this.admitted = plan.admitted;
+ this.reason = plan.reason;
+ this.partitions = plan.partitions;
+ this.descriptorBudget = descriptorBudget;
+ }
+
+ MongoSplitPlan withDescriptorBudget(long bytes) {
+ return new MongoSplitPlan(this, bytes);
+ }
+
+ long descriptorBudget() {
+ return descriptorBudget;
+ }
+
+ static MongoSplitPlan single(int requested, int admitted, String reason) {
+ return new MongoSplitPlan("", "", requested, admitted, reason,
List.of(new BsonDocument()));
+ }
+
+ static MongoSplitPlan ranges(String field, String family, List<BsonValue>
samples, int requested, int admitted) {
+ if (!Set.of("integer", "date", "objectId").contains(family)) {
+ throw new IllegalArgumentException("Unsupported MongoDB split type
family");
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
admitted < 1 || admitted > 256
+ || requested < admitted) {
+ throw new IllegalArgumentException("Invalid MongoDB split plan");
+ }
+ if (admitted == 1 || samples.isEmpty()) {
+ return single(requested, admitted, admitted == 1
+ ? (requested == 1 ? "requested_one" : "admission_single")
: "empty_sample");
+ }
+ var values = new ArrayList<>(samples);
+ Comparator<BsonValue> comparator = comparator(family);
+ values.sort(comparator);
+ List<BsonValue> boundaries = new ArrayList<>();
+ // _id always exists and cannot be an array. Expression comparisons
span the complete BSON
+ // order, including unsampled types; ordinary query comparisons
instead bracket by type.
+ // Custom fields can be missing or multikey, so retain their typed
ranges and complement.
+ boolean wholeId = "_id".equals(field);
+ int ranges = wholeId ? admitted : admitted - 1;
+ for (int j = 1; j < ranges; j++) {
+ BsonValue value = values.get((int) ((long) j * values.size() /
ranges));
+ if (comparator.compare(value, values.get(0)) > 0
+ && (wholeId || comparator.compare(value,
values.get(values.size() - 1)) < 0)
+ && (boundaries.isEmpty() || comparator.compare(value,
boundaries.get(boundaries.size() - 1)) > 0)) {
+ boundaries.add(value);
+ }
+ }
+ // Without an internal bound, typed ranges plus their complement add a
full scan
+ // without splitting the useful key space. One complete cursor
preserves all BSON types.
+ if (boundaries.isEmpty()) {
+ return single(requested, admitted, "insufficient_boundaries");
+ }
+ var types = "integer".equals(family) ? List.of("int", "long") :
List.of(family);
+ var eligibility = new BsonDocument("$in", new BsonArray(List.of(
+ new BsonDocument("$type", new BsonString("$" + field)),
+ new BsonArray(types.stream().map(BsonString::new).toList()))));
+ List<BsonDocument> partitions = new ArrayList<>();
+ for (int i = 0; i <= boundaries.size(); i++) {
+ if (wholeId) {
Review Comment:
The `_id` design rests on this branch: dropping the typed complement is only
a win if MongoDB answers `$expr: {$gte/$lt: ["$_id", <const>]}` from the `_id`
index. MongoDB's documentation only states that guarantee for the `from`
collection of `$lookup`, and the v4.4 wording explicitly said range operators
in `$expr` cannot use an index
(https://github.com/mongodb/docs/blob/master/source/includes/expr-operators-and-indexes.rst).
If a planner ever picks COLLSCAN here, this turns one collection scan into N.
The benchmark numbers suggest the index is used, but the 24 source-profiled
cases assert `one actual split` for custom fields only. Please add a real
multi-split `_id` case with `docsExamined == ROWS` to `verify_native_splits.py`
so a planner regression is caught.
##########
regression-test/suites/external_table_p0/mongodb/test_mongodb_splits.groovy:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_mongodb_splits", "p0,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableMongoSplitTest")))
{
Review Comment:
Neither this flag nor `enableMongoReadParameterTest` (used by
`test_mongodb_read_parameters`) is set in
`regression-test/conf/regression-conf.groovy` or
`pipeline/external/conf/regression-conf.groovy` - they appear only in the
docker README. So no CI job runs the coverage/disjointness/precision assertions
this PR adds; the "all four split SQL suites pass" evidence is manual.
This matters more than usual here: `enable_external_table_batch_mode`
defaults to true, and this is the first PR that makes the MongoDB provider
return >= 0 from `streamingSplitEstimate`, so `auto`/`range` scans take the
`SplitAssignment` + `TSplitSource` + `streamSplits` path for the first time.
The four native suites run with `strategy=single` (which returns -1) and never
touch it. Please enable both flags in the External pipeline conf, which does
start the MongoDB component.
##########
fe/fe-connector/fe-connector-mongodb/src/test/java/org/apache/doris/connector/mongodb/MongoSplitPlanTest.java:
##########
@@ -0,0 +1,284 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.bson.BsonArray;
+import org.bson.BsonDateTime;
+import org.bson.BsonDocument;
+import org.bson.BsonInt32;
+import org.bson.BsonInt64;
+import org.bson.BsonObjectId;
+import org.bson.BsonValue;
+import org.bson.types.ObjectId;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+class MongoSplitPlanTest {
+ @Test
+ void integerBoundariesPreserveSignedWidthsAndUnboundedEnds() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt32(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 4, 4);
+ Assertions.assertEquals(4, plan.size());
+ var first = plan.partition(0).getArray("$and");
+ Assertions.assertEquals("int",
first.get(0).asDocument().getDocument("$expr")
+
.getArray("$in").get(1).asArray().get(0).asString().getValue());
+
Assertions.assertFalse(first.get(1).asDocument().getDocument("sequence").containsKey("$gte"));
+
Assertions.assertFalse(plan.partition(2).getArray("$and").get(1).asDocument()
+ .getDocument("sequence").containsKey("$lt"));
+
Assertions.assertTrue(plan.partition(3).getDocument("$expr").containsKey("$not"));
+ for (long value : new long[] {Long.MIN_VALUE, -2, -1, 0,
9007199254740992L,
+ 9007199254740993L, 9007199254740994L, Long.MAX_VALUE}) {
+ Assertions.assertEquals(1, matches(plan, new BsonInt64(value)),
Long.toString(value));
+ }
+ }
+
+ @Test
+ void duplicateQuantilesKeepFrequencyButNeverPublishDuplicateBoundaries() {
+ var samples = new ArrayList<BsonValue>();
+ for (int i = 0; i < 100; i++) {
+ samples.add(new BsonInt32(i < 80 ? 0 : i));
+ }
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 16,
8);
+ Assertions.assertTrue(plan.size() < 8);
+ for (long value = -1; value <= 101; value++) {
+ Assertions.assertEquals(1, matches(plan, new BsonInt64(value)));
+ }
+ samples.clear();
+ Assertions.assertEquals(1, matches(plan, new BsonInt32(81)));
+ }
+
+ @Test
+ void dateMillisecondsAreIndependentOfSqlPrecisionAndTimezone() {
+ var samples = List.<BsonValue>of(new BsonDateTime(Long.MIN_VALUE), new
BsonDateTime(-1),
+ new BsonDateTime(0), new BsonDateTime(1710064800123L), new
BsonDateTime(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("observed", "date", samples, 5, 5);
+ for (long value : new long[] {Long.MIN_VALUE, -2, -1, 0, 1,
1710064800122L,
+ 1710064800123L, 1710064800124L, Long.MAX_VALUE}) {
+ Assertions.assertEquals(1, matches(plan, new BsonDateTime(value)));
+ }
+ }
+
+ @Test
+ void objectIdsUseAllUnsignedBytes() {
+ var values = List.<BsonValue>of(new BsonObjectId(new
ObjectId("000000000000000000000000")),
+ new BsonObjectId(new ObjectId("7fffffffffffffffffffffff")),
+ new BsonObjectId(new ObjectId("800000000000000000000000")),
+ new BsonObjectId(new ObjectId("ffffffffffffffffffffffff")));
+ var plan = MongoSplitPlan.ranges("_id", "objectId", values, 4, 4);
+ for (BsonValue value : values) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+
+ @Test
+ void complementTestsTheValueItselfInsteadOfArrayElements() {
+ var plan = MongoSplitPlan.ranges("sequence", "integer", List.of(new
BsonInt32(0),
+ new BsonInt64(10), new BsonInt32(20)), 4, 4);
+ var expression =
plan.partition(0).getArray("$and").get(0).asDocument().getDocument("$expr");
+ Assertions.assertEquals(new org.bson.BsonString("$sequence"),
+ expression.getArray("$in").get(0).asDocument().get("$type"));
+ // Missing, null, arrays (even with integer members), double and
Decimal128 are NOT E.
+ for (BsonValue value : List.of(org.bson.BsonNull.VALUE, new
BsonArray(List.of(new BsonInt32(5))),
+ new org.bson.BsonDouble(5), new org.bson.BsonDecimal128(new
org.bson.types.Decimal128(
+ new
java.math.BigDecimal("5.000000000000000000000000000000001"))))) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+
+ @Test
+ void randomizedDuplicateAndSkewedSamplesCoverTheWholeIntegerDomain() {
+ var random = new Random(934351L);
+ for (int trial = 0; trial < 200; trial++) {
+ var values = new ArrayList<BsonValue>();
+ for (int i = 0; i < 1 + random.nextInt(150); i++) {
+ values.add(new BsonInt64(random.nextInt(7) == 0 ?
random.nextLong() : random.nextInt(5)));
+ }
+ int admitted = 2 + random.nextInt(31);
+ var plan = MongoSplitPlan.ranges("sequence", "integer", values,
32, admitted);
+ Assertions.assertTrue(plan.size() <= admitted);
+ for (int i = 0; i < 100; i++) {
+ Assertions.assertEquals(1, matches(plan, new
BsonInt64(random.nextLong())));
+ }
+ }
+ }
+
+ @Test
+ void oneRequestedSplitAndEmptySamplesNeverMeanEmptyResults() {
+ Assertions.assertEquals(new BsonDocument(),
MongoSplitPlan.ranges("sequence", "integer",
+ List.of(new BsonInt32(3)), 1, 1).partition(0));
+ Assertions.assertEquals(new BsonDocument(),
MongoSplitPlan.ranges("sequence", "integer",
+ List.of(), 8, 8).partition(0));
+ }
+
+ @Test
+ void partitionsAreOwnedAndUnsupportedFamiliesFail() {
+ var plan = MongoSplitPlan.ranges("sequence", "integer",
+ List.of(new BsonInt32(1), new BsonInt32(2), new BsonInt32(3)),
3, 3);
+ plan.partition(0).clear();
+ Assertions.assertFalse(plan.partition(0).isEmpty());
+ for (String family : List.of("decimal", "double", "string", "array",
"timestamp")) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> MongoSplitPlan.ranges("sequence", family,
List.of(new BsonInt32(1)), 4, 4));
+ }
+ }
+
+ @Test
+ void idRangesUseTotalOrderExpressionsInsteadOfAComplement() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("_id", "integer", samples, 4, 4);
+ Assertions.assertEquals(4, plan.size());
+ for (int i = 0; i < plan.size(); i++) {
+ var expression = plan.partition(i).get("$expr");
+ Assertions.assertNotNull(expression, "Use expression comparisons
across BSON types");
+
Assertions.assertFalse(expression.asDocument().containsKey("$not"));
+
Assertions.assertFalse(plan.partition(i).toJson().contains("$type"));
+ }
+ Assertions.assertEquals(new BsonInt64(9007199254740993L),
plan.partition(1)
+
.getDocument("$expr").getArray("$and").get(1).asDocument().getArray("$lt").get(1));
+ }
+
+ @Test
+ void oneIdBoundaryValueUsesACompleteSingleScan() {
+ var plan = MongoSplitPlan.ranges("_id", "integer",
+ List.of(new BsonInt32(7), new BsonInt64(7)), 16, 16);
+ Assertions.assertEquals(1, plan.size());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+
+ @Test
+ void idRangesPreserveDateMillisecondsAndSignedIntegerEndpoints() {
+ for (String family : List.of("date", "integer")) {
+ var values = new ArrayList<BsonValue>();
+ for (long value : new long[] {Long.MIN_VALUE, -1, 0,
9007199254740993L, Long.MAX_VALUE}) {
+ values.add(family.equals("date") ? new BsonDateTime(value) :
new BsonInt64(value));
+ }
+ for (int admitted : new int[] {2, 3, 8, 32, 256}) {
+ var plan = MongoSplitPlan.ranges("_id", family, values, 256,
admitted);
+ Assertions.assertTrue(plan.size() <= admitted);
+ for (BsonValue value : values) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+ }
+ }
+
+ @Test
+ void customFieldWithOnlyOneAdmittedRangeUsesACompleteSingleScan() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 8, 2);
+ Assertions.assertEquals(1, plan.size());
+ Assertions.assertEquals("insufficient_boundaries", plan.reason());
+ Assertions.assertEquals("", plan.family());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+
+ @Test
+ void identicalCustomFieldSamplesNeverPublishTwoUnboundedTypeScans() {
+ for (BsonValue value : List.of(new BsonInt64(9007199254740993L),
+ new BsonDateTime(Long.MIN_VALUE), new BsonObjectId(new
ObjectId("800000000000000000000001")))) {
+ String family = value.isInt64() ? "integer" : value.isDateTime() ?
"date" : "objectId";
+ for (int admitted : new int[] {2, 3, 8, 256}) {
+ var plan = MongoSplitPlan.ranges("sequence", family,
List.of(value, value, value), 256, admitted);
+ Assertions.assertEquals(1, plan.size(), family + ":" +
admitted);
+ Assertions.assertEquals("insufficient_boundaries",
plan.reason());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+ }
+ }
+
+ @Test
+ void encodedPartitionsAreExactAndOwnedByEachDescriptor() {
+ for (String family : List.of("integer", "date", "objectId")) {
+ List<BsonValue> values = switch (family) {
+ case "integer" -> List.of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(9007199254740993L),
+ new BsonInt64(Long.MAX_VALUE));
+ case "date" -> List.of(new BsonDateTime(Long.MIN_VALUE), new
BsonDateTime(-1),
+ new BsonDateTime(Long.MAX_VALUE));
+ default -> List.of(new BsonObjectId(new
ObjectId("000000000000000000000001")),
+ new BsonObjectId(new
ObjectId("800000000000000000000001")),
+ new BsonObjectId(new
ObjectId("ffffffffffffffffffffffff")));
+ };
+ var plan = MongoSplitPlan.ranges("_id", family, values, 3, 3);
+ var budgeted = plan.withDescriptorBudget(1024);
+ for (int index = 0; index < plan.size(); index++) {
+ byte[] expected = MongoSplitPlan.encode(plan.partition(index));
+ byte[] exposed = budgeted.partitionBytes(index);
+ Assertions.assertArrayEquals(expected, exposed);
+ java.util.Arrays.fill(exposed, (byte) 0);
+ Assertions.assertArrayEquals(expected,
plan.partitionBytes(index));
+ Assertions.assertArrayEquals(expected,
budgeted.partitionBytes(index));
+ Assertions.assertArrayEquals(expected,
MongoSplitPlan.encode(plan.partition(index)));
+ }
+ }
+ }
+
+ private static int matches(MongoSplitPlan plan, BsonValue value) {
+ boolean scalar = switch (plan.family()) {
+ case "integer" -> value.isInt32() || value.isInt64();
+ case "date" -> value.isDateTime();
+ case "objectId" -> value.isObjectId();
+ default -> false;
+ };
+ int matches = 0;
+ for (int i = 0; i < plan.size(); i++) {
+ BsonDocument partition = plan.partition(i);
+ if ("_id".equals(plan.field()) && partition.containsKey("$expr")) {
+ var expression = partition.getDocument("$expr");
+ List<BsonValue> comparisons = expression.containsKey("$and")
+ ? expression.getArray("$and").getValues() :
List.of(expression);
+ boolean included = true;
+ for (BsonValue comparison : comparisons) {
+ var predicate = comparison.asDocument();
+ boolean lower = predicate.containsKey("$gte");
+ var operands = predicate.getArray(lower ? "$gte" : "$lt");
+ int order = compare(value, operands.get(1));
+ included &= lower ? order >= 0 : order < 0;
+ }
+ if (included) {
+ matches++;
+ }
+ } else if (partition.isEmpty() || (partition.containsKey("$expr")
&& !scalar)) {
+ matches++;
+ } else if (partition.containsKey("$and") && scalar) {
+ var interval = partition.getArray("$and").get(1).asDocument();
+ var bounds = interval.getDocument(plan.field(), new
BsonDocument());
+ if ((!bounds.containsKey("$gte") || compare(value,
bounds.get("$gte")) >= 0)
+ && (!bounds.containsKey("$lt") || compare(value,
bounds.get("$lt")) < 0)) {
+ matches++;
+ }
+ }
+ }
+ return matches;
+ }
+
+ private static int compare(BsonValue left, BsonValue right) {
Review Comment:
Because `compare()` goes through `asNumber()` / `asDateTime()` /
`asObjectId()`, a document whose `_id` is an unsampled BSON type (string,
embedded document, ...) throws here. That means the one property which
justifies the `wholeId` design - that `$expr` ranges cover the complete BSON
order - cannot be checked at this layer; it is only covered indirectly by
`mixed_id_records` in the SQL suite, which itself is not enabled in CI.
A small known-order table with one value per BSON type, asserting exactly
one partition matches each, would make this oracle actually test the claim.
##########
fe/fe-connector/fe-connector-mongodb/src/test/java/org/apache/doris/connector/mongodb/MongoRangePlannerTest.java:
##########
@@ -0,0 +1,720 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.connector.spi.scan.ConnectorScanRequest;
+import org.apache.doris.thrift.TFileRangeDesc;
+import org.apache.doris.thrift.TMongoScanDesc;
+import org.apache.doris.thrift.TMongoSnapshot;
+import org.apache.doris.thrift.TTableFormatFileDesc;
+
+import org.bson.BsonInt32;
+import org.bson.BsonValue;
+import org.bson.Document;
+import org.bson.types.Binary;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+class MongoRangePlannerTest {
+ private static class Client extends MongoSnapshotPlanTest.Client {
+ int probes;
+ RuntimeException failure;
+ final List<BsonValue> samples = new ArrayList<>();
+ TMongoSnapshot sampledSnapshot;
+ boolean indexed = true;
+ long logicalBytes = 4L * 1024 * 1024 * 1024;
+ long sampleBytes = 1024;
+
+ Client() {
+ for (int i = 0; i < 100; i++) {
+ samples.add(new BsonInt32(i));
+ }
+ }
+
+ @Override
+ public MongoSplitProbe probeSplits(MongoTableHandle table, String
field, String preference,
+ TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable
checkCancelled) {
+ probes++;
+ sampledSnapshot = snapshot.deepCopy();
+ checkCancelled.run();
+ if (failure != null) {
+ throw failure;
+ }
+ return new MongoSplitProbe(samples, logicalBytes, sampleBytes,
indexed);
+ }
+ }
+
+ private static MongoScanPlanTest.Session session(ConnectorStatementScope
scope, Map<String, String> options) {
+ return new MongoScanPlanTest.Session(options) {
+ @Override
+ public ConnectorStatementScope getStatementScope() {
+ return scope;
+ }
+ };
+ }
+
+ private static MongoTableHandle table(MongoCatalogProperties properties,
String kind) {
+ return new MongoTableHandle("sample", new Document("name",
"records").append("type", kind)
+ .append("info", new Document("uuid", new Binary((byte) 4, new
byte[16]))), properties);
+ }
+
+ private static MongoCatalogProperties properties() {
+ return MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017"));
+ }
+
+ private static Map<String, String> options(String strategy) {
+ return Map.of("mongodb_read_concern", "snapshot",
"mongodb_split_strategy", strategy,
+ "file_split_size", "536870912", "max_file_split_num", "8");
+ }
+
+ private static TMongoScanDesc
descriptor(org.apache.doris.connector.spi.scan.ConnectorScanRange range) {
+ var result = new TTableFormatFileDesc();
+ range.populateRangeParams(result, new TFileRangeDesc());
+ return result.getMongodbParams();
+ }
+
+ @Test
+ void familyClassifierPreservesExactTypesFrequencyAndDeterministicTies() {
+ var integer = new org.bson.BsonInt64(Long.MIN_VALUE);
+ var date = new org.bson.BsonDateTime(Long.MAX_VALUE);
+ var objectId = new org.bson.BsonObjectId(new
org.bson.types.ObjectId("800000000000000000000000"));
+ Assertions.assertEquals("integer", MongoRangePlanner.family(new
BsonInt32(1)));
+ Assertions.assertEquals("integer", MongoRangePlanner.family(integer));
+ Assertions.assertEquals("date", MongoRangePlanner.family(date));
+ Assertions.assertEquals("objectId",
MongoRangePlanner.family(objectId));
+ for (BsonValue unsupported : List.of(org.bson.BsonNull.VALUE, new
org.bson.BsonDouble(1.0),
+ new
org.bson.BsonDecimal128(org.bson.types.Decimal128.parse("1.000")),
+ new org.bson.BsonString("value"), org.bson.BsonBoolean.TRUE,
new org.bson.BsonArray(),
+ new org.bson.BsonDocument(), new org.bson.BsonTimestamp(1,
1))) {
+ Assertions.assertEquals("", MongoRangePlanner.family(unsupported));
+ }
+ Assertions.assertEquals("integer",
MongoRangePlanner.family(List.of(integer, new BsonInt32(1), date)));
+ Assertions.assertEquals("date",
MongoRangePlanner.family(List.of(integer, date, objectId)));
+ Assertions.assertEquals("date",
MongoRangePlanner.family(List.of(objectId, date, integer)));
+ Assertions.assertEquals("",
MongoRangePlanner.family(List.<BsonValue>of(org.bson.BsonNull.VALUE)));
+ }
+
+ @Test
+ void mutatingWireFilterDoesNotCorruptTheMemoizedPlan() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ var first = descriptor(provider.planScan(session, request).get(0));
+ byte[] expected = first.getSplit().getFilterBson().clone();
+ java.util.Arrays.fill(first.getSplit().getFilterBson(), (byte) 0);
+ var fresh = descriptor(provider.planScan(session, request).get(0));
+ Assertions.assertArrayEquals(expected,
fresh.getSplit().getFilterBson());
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void rangePlanIsFrozenBeforeAnySplitAndExplainReusesIt() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var table = table(properties, "collection");
+ var explanation = provider.getScanNodeProperties(session, table,
List.of(), Optional.empty());
+ var ranges = provider.planScan(session,
ConnectorScanRequest.builder(table, List.of()).build());
+ Assertions.assertEquals(8, ranges.size());
+ Assertions.assertEquals("8", explanation.get("splits_actual"));
+ Assertions.assertEquals(1, client.probes);
+ for (int i = 0; i < ranges.size(); i++) {
+ var d = descriptor(ranges.get(i));
+ Assertions.assertEquals(client.sampledSnapshot, d.getSnapshot());
+ Assertions.assertEquals(2, d.getVersion());
+ Assertions.assertEquals(i, d.getSplit().getIndex());
+ Assertions.assertEquals(8, d.getSplit().getActual());
+ Assertions.assertEquals("integer", d.getSplit().getTypeFamily());
+ }
+ Assertions.assertEquals(8, ranges.stream().map(r ->
r.getSplitIdentity()).distinct().count());
+ client.samples.clear();
+ Assertions.assertEquals(8, provider.planScan(session,
+ ConnectorScanRequest.builder(table,
List.of()).build()).size());
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void sourceErrorsAreStickyAndNeverBecomeFullRescans() {
+ var properties = properties();
+ var client = new Client();
+ client.failure = new IllegalStateException("fixture planning failure");
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("auto"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ Assertions.assertThrows(IllegalStateException.class, () ->
provider.planScan(session, request));
+ client.failure = null;
+ Assertions.assertThrows(IllegalStateException.class, () ->
provider.planScan(session, request));
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void singleStrategyDoesNotProbeAndExplicitUnsupportedObjectsFail() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var single = session(new MongoSnapshotPlanTest.Scope(),
Map.of("mongodb_split_strategy", "single"));
+ Assertions.assertEquals(1, provider.planScan(single,
+ ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build()).size());
+ Assertions.assertEquals(0, client.probes);
+ for (String kind : List.of("view", "timeseries")) {
+ var explicit = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
provider.planScan(explicit,
+ ConnectorScanRequest.builder(table(properties, kind),
List.of()).build()));
+ }
+ }
+
+ @Test
+ void explicitRangeRequiresSnapshotAndLiveScope() {
+ var properties = properties();
+ var provider = new MongoScanPlanProvider(properties, Client::new);
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
provider.planScan(
+ session(new MongoSnapshotPlanTest.Scope(),
Map.of("mongodb_split_strategy", "range")), request));
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> provider.planScan(new
MongoScanPlanTest.Session(options("range")), request));
+ }
+
+ @Test
+ void autoFallbackAndExplicitRejectionRetainDecisionReasons() {
+ for (String condition : List.of("missing_index",
"unknown_complement_cost", "small_collection",
+ "unsupported_type", "probe_budget", "empty_sample")) {
+ var properties = properties();
+ var client = new Client();
+ switch (condition) {
+ case "missing_index" -> client.indexed = false;
+ case "unknown_complement_cost" -> client.logicalBytes = -1;
+ case "small_collection" -> client.logicalBytes = 0;
+ case "unsupported_type" -> {
+ client.samples.clear();
+ client.samples.add(org.bson.BsonNull.VALUE);
+ }
+ case "probe_budget" -> client.failure = new
MongoRangePlanner.PlanningLimit();
+ case "empty_sample" -> client.samples.clear();
+ default -> throw new AssertionError(condition);
+ }
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var table = table(properties, "collection");
+ var request = ConnectorScanRequest.builder(table,
List.of()).build();
+ var auto = session(new MongoSnapshotPlanTest.Scope(),
options("auto"));
+ var range = descriptor(provider.planScan(auto, request).get(0));
+ Assertions.assertEquals("single", range.getSplit().getStrategy());
+ Assertions.assertEquals(condition, range.getSplit().getReason());
+ if (!List.of("unknown_complement_cost", "small_collection",
"empty_sample").contains(condition)) {
+ var explicit = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
provider.planScan(explicit, request));
+ }
+ }
+ }
+
+ @Test
+ void budgetsBoundAllScansOfOneStatementAndSourceAndRefundUnusedRanges() {
+ var budget = new MongoRangePlanner.Budget();
+ Assertions.assertEquals(255, budget.admit("source-a", 255, 1));
+ Assertions.assertEquals(1, budget.admit("source-a", 8, 1));
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> budget.admit("source-a", 1, 1));
+ budget.refund("source-a", 7, 1);
+ Assertions.assertEquals(7, budget.admit("source-a", 8, 1));
+ for (int i = 1; i < 16; i++) {
+ Assertions.assertEquals(256, budget.admit("source-" + i, 256, 1));
+ }
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> budget.admit("source-next", 1, 1));
+ budget.sampled(1024 * 1024);
+ Assertions.assertThrows(MongoRangePlanner.PlanningLimit.class, () ->
budget.sampled(1024 * 1024 + 1));
+ for (int i = 1; i < 16; i++) {
+ budget.sampled(1024 * 1024);
+ }
+ Assertions.assertThrows(MongoRangePlanner.PlanningLimit.class, () ->
budget.sampled(1));
+ }
+
+ @Test
+ void streamingFreezesOnceAndNeverReprobesAfterPublishing() throws
Exception {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ try (var source = provider.streamSplits(session, request)) {
+ Assertions.assertEquals(0, client.probes);
+ Assertions.assertTrue(source.hasNext());
+ var first = descriptor(source.next());
+ client.failure = new IllegalStateException("must never re-probe");
+ Assertions.assertTrue(source.hasNext());
+ var second = descriptor(source.next());
+ Assertions.assertEquals(first.getSnapshot(), second.getSnapshot());
+ Assertions.assertEquals(1, client.probes);
+ source.close();
+ Assertions.assertFalse(source.hasNext());
+ Assertions.assertThrows(java.util.NoSuchElementException.class,
source::next);
+ }
+ }
+
+ @Test
+ void customFieldDoesNotSilentlyApplyToMultipleScans() {
+ var properties = properties();
+ var client = new Client();
+ var scope = new MongoSnapshotPlanTest.Scope();
+ var session = session(scope, Map.of("mongodb_read_concern",
"snapshot", "mongodb_split_strategy", "range",
+ "max_file_split_num", "8", "mongodb_split_field", "sequence"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ var first = new MongoScanPlanProvider(properties, () -> client);
+ first.planScan(session, request);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new MongoScanPlanProvider(properties, () ->
client).planScan(session, request));
+ }
+
+ @Test
+ void unsafeFieldsAndMalformedPoliciesFailBeforeProbing() {
+ var properties = properties();
+ var client = new Client();
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ for (var setting : Map.of("file_split_size", "invalid",
"max_file_split_size", "0",
+ "max_file_split_num", "not-a-number",
"max_initial_file_split_size", "-1",
+ "mongodb_split_field", "nested.value",
"mongodb_split_strategy", "invalid").entrySet()) {
+ var options = new java.util.HashMap<>(options("range"));
+ options.put(setting.getKey(), setting.getValue());
+ if (setting.getKey().equals("max_file_split_size")
+ || setting.getKey().equals("max_initial_file_split_size"))
{
+ options.put("file_split_size", "0");
+ }
+ Assertions.assertThrows(IllegalArgumentException.class, () -> new
MongoScanPlanProvider(properties,
+ () -> client).planScan(session(new
MongoSnapshotPlanTest.Scope(), options), request));
+ }
+ Assertions.assertEquals(0, client.probes);
+ }
+
+ @Test
+ void oneProviderStillPlansDistinctNamespacesIndependently() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ provider.planScan(session,
ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build());
+ var another = new MongoTableHandle("sample", new Document("name",
"another_records")
+ .append("type", "collection").append("info", new
Document("uuid", new Binary((byte) 4, new byte[16]))),
+ properties);
+ provider.planScan(session, ConnectorScanRequest.builder(another,
List.of()).build());
+ Assertions.assertEquals(2, client.probes);
+ }
+
+ @Test
+ void autoCanFallBackBeforePublicationWhenAllProbeSlotsAreBusy() throws
Exception {
+ var properties = properties();
+ var entered = new java.util.concurrent.CountDownLatch(2);
+ var release = new java.util.concurrent.CountDownLatch(1);
+ var blocked = new Client() {
+ @Override
+ public MongoSplitProbe probeSplits(MongoTableHandle table, String
field, String preference,
+ TMongoSnapshot snapshot, int rows, long timeoutMs,
Runnable checkCancelled) {
+ entered.countDown();
+ try {
+ if (!release.await(5,
java.util.concurrent.TimeUnit.SECONDS)) {
+ throw new AssertionError("Probe fixture was not
released");
+ }
+ } catch (InterruptedException interrupted) {
+ Thread.currentThread().interrupt();
+ throw new CancellationException();
+ }
+ return super.probeSplits(table, field, preference, snapshot,
rows, timeoutMs, checkCancelled);
+ }
+ };
+ var scope = new MongoSnapshotPlanTest.Scope();
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ var executor = java.util.concurrent.Executors.newFixedThreadPool(2);
+ try {
+ var first = executor.submit(() -> new
MongoScanPlanProvider(properties, () -> blocked)
+ .planScan(session(scope, options("range")), request));
+ var second = executor.submit(() -> new
MongoScanPlanProvider(properties, () -> blocked)
+ .planScan(session(scope, options("range")), request));
+ Assertions.assertTrue(entered.await(5,
java.util.concurrent.TimeUnit.SECONDS));
+ var policy = new java.util.HashMap<>(options("auto"));
+ // The optional probe budget is internal; waiting for a probe slot
consumes it too.
+ var plan = new MongoScanPlanProvider(properties,
Client::new).planScan(session(scope, policy), request);
+ Assertions.assertEquals("probe_budget",
descriptor(plan.get(0)).getSplit().getReason());
+ release.countDown();
+ Assertions.assertEquals(8, first.get(5,
java.util.concurrent.TimeUnit.SECONDS).size());
+ Assertions.assertEquals(8, second.get(5,
java.util.concurrent.TimeUnit.SECONDS).size());
+ } finally {
+ release.countDown();
+ executor.shutdownNow();
+ Assertions.assertTrue(executor.awaitTermination(5,
java.util.concurrent.TimeUnit.SECONDS));
+ }
+ }
+
+ @Test
+ void explicitFieldRestrictionAlsoCountsAutoFallbackScans() {
+ var properties = properties();
+ var scope = new MongoSnapshotPlanTest.Scope();
+ var session = session(scope, Map.of("mongodb_read_concern",
"majority", "mongodb_split_strategy", "auto",
+ "max_file_split_num", "8", "mongodb_split_field", "sequence"));
+ new MongoScanPlanProvider(properties, Client::new).planScan(session,
+ ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build());
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> new MongoScanPlanProvider(properties,
Client::new).planScan(session,
+ ConnectorScanRequest.builder(table(properties,
"view"), List.of()).build()));
+ }
+
+ @Test
+ void wideDescriptorsCannotMultiplyIntoAnOversizedSplitRpc() throws
Exception {
+ var properties = MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017",
+ "mongodb.username", "fixture_reader", "mongodb.password",
"fixture测".repeat(200000)));
+ var client = new Client();
+ var policy = new java.util.HashMap<>(options("range"));
+ policy.put("max_file_split_num", "256");
+ policy.put("file_split_size", "1");
+ var ranges = new MongoScanPlanProvider(properties, () ->
client).planScan(
+ session(new MongoSnapshotPlanTest.Scope(), policy),
+ ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build());
+ var first = descriptor(ranges.get(0));
+ int encoded = new org.apache.thrift.TSerializer(new
org.apache.thrift.protocol.TBinaryProtocol.Factory())
+ .serialize(first).length;
+ Assertions.assertTrue((long) encoded * ranges.size() <= 8L * 1024 *
1024,
+ "Repeated descriptor metadata must fit within the per-plan RPC
budget");
+ Assertions.assertEquals(256, first.getSplit().getRequested());
+ Assertions.assertTrue(first.getSplit().getAdmitted() < 256);
+ }
+
+ @Test
+ void
descriptorByteAdmissionIsSharedBySourceAndStatementAndRefundedExactly() {
+ var budget = new MongoRangePlanner.Budget();
+ long megabyte = 1024 * 1024;
+ Assertions.assertEquals(32, budget.admit("source-a", 256, megabyte));
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> budget.admit("source-a", 1, 1));
+ budget.refund("source-a", 16, megabyte);
+ Assertions.assertEquals(16, budget.admit("source-a", 32, megabyte));
+ Assertions.assertEquals(32, budget.admit("source-b", 256, megabyte));
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> budget.admit("source-c", 1, 1));
+ budget.refund("source-b", 1, megabyte);
+ Assertions.assertEquals(1, budget.admit("source-c", 8, megabyte));
+ }
+
+ @Test
+ void
candidateGrowthAfterPublicationFailsInsteadOfBypassingAdmissionOrReplanning() {
+ var column = new Document("name", "text").append("path",
List.of("text"))
+ .append("doris_type", "STRING").append("bson_types",
List.of("string"));
+ var properties = MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017",
+ "mongodb.type_mismatch", "null", "mongodb.schema",
+ new Document("sample.records", new Document("columns",
List.of(column))).toJson()));
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var policy = new java.util.HashMap<>(options("range"));
+ policy.put("max_file_split_num", "32");
+ policy.put("file_split_size", "1");
+ var session = session(new MongoSnapshotPlanTest.Scope(), policy);
+ var table = table(properties, "collection");
+ provider.planScan(session, ConnectorScanRequest.builder(table,
List.of()).build()).get(0);
+ var filter = new
org.apache.doris.connector.spi.pushdown.ConnectorComparison(
+
org.apache.doris.connector.spi.pushdown.ConnectorComparison.Operator.EQ,
+ new
org.apache.doris.connector.spi.pushdown.ConnectorColumnRef("text",
+
org.apache.doris.connector.spi.ConnectorType.of("STRING")),
+
org.apache.doris.connector.spi.pushdown.ConnectorLiteral.ofString("v".repeat(60000)));
+ Assertions.assertEquals(MongoPredicateTranslator.Kind.SUPERSET,
+ new MongoPredicateTranslator(table.getSchema(),
true).translate(filter).kind());
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> provider.planScan(session,
ConnectorScanRequest.builder(table,
List.of()).filter(Optional.of(filter)).build())
+ .get(0));
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void cancelledPlanningPublishesNothing() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var context = new ConnectorScanContext();
+ context.cancel();
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of())
+ .scanContext(context).build();
+ Assertions.assertThrows(CancellationException.class, () ->
provider.planScan(
+ session(new MongoSnapshotPlanTest.Scope(), options("range")),
request));
+ Assertions.assertEquals(0, client.probes);
+ }
+
+ @Test
+ void commonSplitSizeAndNumberLimitDriveRangePlanning() {
+ var properties = properties();
+ var client = new Client();
+ var policy = Map.of("mongodb_read_concern", "snapshot",
"mongodb_split_strategy", "range",
+ "file_split_size", "536870912", "max_file_split_num", "3");
+ var ranges = new MongoScanPlanProvider(properties, () ->
client).planScan(
+ session(new MongoSnapshotPlanTest.Scope(), policy),
+ ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build());
+ Assertions.assertEquals(3, ranges.size());
+ Assertions.assertEquals(3,
descriptor(ranges.get(0)).getSplit().getRequested());
+ }
+
+ @Test
+ void propertiesDoNotSelectSnapshotOrProbeBeforeTheFinalRequest() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var table = table(properties, "collection");
+ var explanation = provider.getScanNodePropertiesResult(session, table,
List.of(), Optional.empty())
+ .getProperties();
+ Assertions.assertEquals(0, client.identities);
+ Assertions.assertEquals(0, client.selections);
+ Assertions.assertEquals(0, client.probes);
+ provider.planScan(session, ConnectorScanRequest.builder(table,
List.of()).build()).get(0);
+ Assertions.assertEquals("1700000000:7", explanation.get("snapshot"));
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void ordinaryLargePredicateIsAdmittedAfterEarlyProperties() {
+ var column = new Document("name", "text").append("path",
List.of("text"))
+ .append("doris_type", "STRING").append("bson_types",
List.of("string"));
+ var properties = MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017",
+ "mongodb.type_mismatch", "null", "mongodb.schema",
+ new Document("sample.records", new Document("columns",
List.of(column))).toJson()));
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var table = table(properties, "collection");
+ provider.getScanNodeProperties(session, table, List.of(),
Optional.empty());
+ var filter = new
org.apache.doris.connector.spi.pushdown.ConnectorComparison(
+
org.apache.doris.connector.spi.pushdown.ConnectorComparison.Operator.EQ,
+ new
org.apache.doris.connector.spi.pushdown.ConnectorColumnRef("text",
+
org.apache.doris.connector.spi.ConnectorType.of("STRING")),
+
org.apache.doris.connector.spi.pushdown.ConnectorLiteral.ofString("v".repeat(60000)));
+ Assertions.assertEquals(MongoPredicateTranslator.Kind.SUPERSET,
+ new MongoPredicateTranslator(table.getSchema(),
true).translate(filter).kind());
+ var ranges = provider.planScan(session,
+ ConnectorScanRequest.builder(table,
List.of()).filter(Optional.of(filter)).build());
+ Assertions.assertTrue(descriptor(ranges.get(0)).getFilterBson().length
> 60000);
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void emptySampleFallbacksCannotEscapePinnedSourcePartitionLimit() {
+ var properties = properties();
+ var client = new Client();
+ client.samples.clear();
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ for (int index = 0; index < 256; index++) {
+ var ranges = new MongoScanPlanProvider(properties, () ->
client).planScan(session, request);
+ Assertions.assertEquals(1, ranges.size());
+ }
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> new MongoScanPlanProvider(properties, () ->
client).planScan(session, request));
+ }
+
+ @Test
+ void successfulFallbackDescriptorsStillConsumeSourceByteBudget() {
+ var properties = MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017",
+ "mongodb.username", "fixture_reader", "mongodb.password",
"p".repeat(1024 * 1024)));
+ var client = new Client();
+ client.samples.clear();
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ for (int index = 0; index < 31; index++) {
+ Assertions.assertEquals(1, new MongoScanPlanProvider(properties,
() -> client)
+ .planScan(session, request).size());
+ }
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> new MongoScanPlanProvider(properties, () ->
client).planScan(session, request));
+ }
+
+ @Test
+ void sharedSizeHeuristicsAndNonpositiveUserCapsKeepTheirExistingMeaning() {
+ var scope = new MongoSnapshotPlanTest.Scope();
+ var defaults = session(scope, Map.of());
+ Assertions.assertEquals(32L * 1024 * 1024,
MongoRangePlanner.targetBytes(defaults, 1024));
+ Assertions.assertEquals(64L * 1024 * 1024,
+ MongoRangePlanner.targetBytes(defaults, 64L * 1024 * 1024 *
200));
+ Assertions.assertEquals(64L * 1024 * 1024,
MongoRangePlanner.targetBytes(defaults, -1));
+ Assertions.assertEquals(256, MongoRangePlanner.splitLimit(defaults));
+ for (String unlimited : List.of("0", "-1")) {
+ Assertions.assertEquals(256, MongoRangePlanner.splitLimit(
+ session(scope, Map.of("max_file_split_num", unlimited))));
+ }
+ var huge = session(scope, Map.of("file_split_size",
Long.toString(Long.MAX_VALUE)));
+ Assertions.assertEquals(Long.MAX_VALUE,
MongoRangePlanner.targetBytes(huge, Long.MAX_VALUE));
+ var threshold = session(scope, Map.of("max_file_split_size",
Long.toString(Long.MAX_VALUE),
+ "max_initial_file_split_size", "1048576",
"max_initial_file_split_num", "200"));
+ Assertions.assertEquals(1048576,
MongoRangePlanner.targetBytes(threshold, Long.MAX_VALUE));
+ }
+
+ @Test
+ void singleStrategyRemainsExemptButEveryAutoFallbackIsAdmitted() {
+ var properties = properties();
+ var client = new Client();
+ var scope = new MongoSnapshotPlanTest.Scope();
+ var single = session(scope, Map.of("mongodb_split_strategy",
"single"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ for (int index = 0; index < 300; index++) {
+ Assertions.assertEquals(1, new MongoScanPlanProvider(properties,
() -> client)
+ .planScan(single, request).size());
+ }
+ var auto = session(scope, Map.of("mongodb_split_strategy", "auto",
"mongodb_read_concern", "majority"));
+ for (int index = 0; index < 256; index++) {
+ Assertions.assertEquals(1, new MongoScanPlanProvider(properties,
() -> client)
+ .planScan(auto, request).size());
+ }
+
Assertions.assertThrows(org.apache.doris.connector.spi.DorisConnectorException.class,
+ () -> new MongoScanPlanProvider(properties, () ->
client).planScan(auto, request));
+ Assertions.assertEquals(0, client.probes);
+ }
+
+ @Test
+ void offlineAutoFallbackNeedsNoPinnedSnapshotOrQueryDeadline() {
+ var properties = properties();
+ var client = new Client();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var session = session(ConnectorStatementScope.NONE,
+ Map.of("mongodb_split_strategy", "auto",
"mongodb_read_concern", "majority"));
+ var ranges = provider.planScan(session,
+ ConnectorScanRequest.builder(table(properties, "collection"),
List.of()).build());
+ Assertions.assertEquals(1, ranges.size());
+ Assertions.assertEquals("snapshot_required",
descriptor(ranges.get(0)).getSplit().getReason());
+ Assertions.assertEquals(0, client.probes);
+ }
+
+ @Test
+ void repeatedTimeseriesFallbacksAdmitTheirCompleteSnapshotPayload() throws
Exception {
+ var columns = new ArrayList<Document>();
+ for (int i = 0; i < 900; i++) {
+ columns.add(new Document("name", String.format("c%04d",
i)).append("path", List.of("v"))
+ .append("bson_types",
List.of("long")).append("doris_type", "BIGINT"));
+ }
+ var properties = MongoCatalogProperties.of(Map.of("mongodb.uri",
"mongodb://localhost:27017",
+ "mongodb.schema", new Document("sample.records", new
Document("columns", columns)).toJson()));
+ var table = new MongoTableHandle("sample", new Document("name",
"records").append("type", "timeseries")
+ .append("options", new Document("timeseries", new
Document("timeField", "t".repeat(15000)))),
+ properties);
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("auto"));
+ var request = ConnectorScanRequest.builder(table, List.of()).build();
+ var serializer = new org.apache.thrift.TSerializer(new
org.apache.thrift.protocol.TBinaryProtocol.Factory());
+ long wireBytes = 0;
+ boolean limited = false;
+ for (int i = 0; i < 256; i++) {
+ try {
+ var ranges = new MongoScanPlanProvider(properties,
Client::new).planScan(session, request);
+ wireBytes +=
serializer.serialize(descriptor(ranges.get(0))).length;
+ Assertions.assertTrue(wireBytes <= 32L * 1024 * 1024,
+ "Serialized fallback descriptors exceeded the
pinned-source byte budget");
+ } catch (org.apache.doris.connector.spi.DorisConnectorException
limit) {
+ Assertions.assertTrue(limit.getMessage().contains("budget
exhausted"));
+ limited = true;
+ break;
+ }
+ }
+ Assertions.assertTrue(limited);
+ }
+
+ @Test
+ void customFieldWithoutInternalBoundsPublishesOneSnapshotDescriptor() {
+ var properties = properties();
+ var client = new Client();
+ var policy = new java.util.HashMap<>(options("range"));
+ policy.put("max_file_split_num", "2");
+ policy.put("mongodb_split_field", "sequence");
+ var session = session(new MongoSnapshotPlanTest.Scope(), policy);
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var table = table(properties, "collection");
+ var ranges = provider.planScan(session,
ConnectorScanRequest.builder(table, List.of()).build());
+ Assertions.assertEquals(1, ranges.size());
+ var scan = descriptor(ranges.get(0));
+ Assertions.assertEquals(1, scan.getVersion());
+ Assertions.assertEquals("single", scan.getSplit().getStrategy());
+ Assertions.assertEquals("insufficient_boundaries",
scan.getSplit().getReason());
+ Assertions.assertEquals(client.time.getTime(),
scan.getSnapshot().getTimestampSeconds());
+ Assertions.assertEquals("1", provider.getScanNodeProperties(session,
table, List.of(), Optional.empty())
+ .get("splits_actual"));
+ Assertions.assertEquals(1, client.probes);
+ }
+
+ @Test
+ void assertionErrorCompletesTheSharedRangePlanForEveryWaiter() throws
Exception {
+ assertFatalRangeFailure(new AssertionError("controlled range
failure"));
+ }
+
+ @Test
+ void outOfMemoryErrorCompletesTheSharedRangePlanForEveryWaiter() throws
Exception {
+ assertFatalRangeFailure(new OutOfMemoryError("controlled range
failure"));
+ }
+
+ private static void assertFatalRangeFailure(Error fatal) throws Exception {
+ var entered = new CountDownLatch(1);
+ var release = new CountDownLatch(1);
+ var waiting = new CountDownLatch(1);
+ var loads = new AtomicInteger();
+ var client = new Client() {
+ @Override
+ public MongoSplitProbe probeSplits(MongoTableHandle table, String
field, String preference,
+ TMongoSnapshot snapshot, int rows, long timeoutMs,
Runnable checkCancelled) {
+ loads.incrementAndGet();
+ entered.countDown();
+ try {
+ Assertions.assertTrue(release.await(5, TimeUnit.SECONDS));
+ } catch (InterruptedException interrupted) {
+ Thread.currentThread().interrupt();
+ throw new CancellationException();
+ }
+ throw fatal;
+ }
+ };
+ var properties = properties();
+ var session = session(new MongoSnapshotPlanTest.Scope(),
options("range"));
+ var request = ConnectorScanRequest.builder(table(properties,
"collection"), List.of()).build();
+ var provider = new MongoScanPlanProvider(properties, () -> client);
+ var executor = Executors.newFixedThreadPool(2);
+ try {
+ var owner = executor.submit(() -> provider.planScan(session,
request));
+ Assertions.assertTrue(entered.await(5, TimeUnit.SECONDS));
+ var waiter = executor.submit(() -> {
+ waiting.countDown();
+ return provider.planScan(session, request);
+ });
+ Assertions.assertTrue(waiting.await(5, TimeUnit.SECONDS));
+ release.countDown();
+ Assertions.assertSame(fatal,
Assertions.assertThrows(ExecutionException.class,
+ () -> owner.get(1, TimeUnit.SECONDS)).getCause());
Review Comment:
1-second futures on a shared CI host are a flake risk (one JIT/GC pause
turns a pass into a timeout), and
`autoCanFallBackBeforePublicationWhenAllProbeSlotsAreBusy` also really waits
out the 2s `PROBE_TIMEOUT_MS`, adding fixed time to this class. Please use a
generous timeout here and, for the probe-budget case, an injectable smaller
budget rather than sleeping through the production constant.
##########
regression-test/conf/regression-conf.groovy:
##########
@@ -286,7 +286,10 @@ trino_jdbc_port = 8080
// trino-connector catalog test config
enableTrinoConnectorTest = false
-// Optional MongoDB fixture: run-thirdparties-docker.sh -c mongodb.
+// Native coverage must not depend on installing the optional Trino MongoDB
plugin.
+// Start the fixture with run-thirdparties-docker.sh -c mongodb.
+enableMongoDBNativeTest = true
Review Comment:
This is the shared template used by local runs and by any pipeline that does
not override it, and it enables four native suites that do not exercise the
range/split feature at all. Every other optional external connector here
defaults to false (`enableHiveTest=false`, `enableJdbcTest=false`,
`enableMongoDBTest=false`), and the MongoDB fixture is normally not running in
a local/P0 run, so these suites fail at `CREATE CATALOG`.
The External pipeline already sets this in
`pipeline/external/conf/regression-conf.groovy`. Please keep the root default
false and enable it only where the fixture is started.
##########
fe/fe-connector/fe-connector-mongodb/src/test/java/org/apache/doris/connector/mongodb/MongoSplitPlanTest.java:
##########
@@ -0,0 +1,284 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.bson.BsonArray;
+import org.bson.BsonDateTime;
+import org.bson.BsonDocument;
+import org.bson.BsonInt32;
+import org.bson.BsonInt64;
+import org.bson.BsonObjectId;
+import org.bson.BsonValue;
+import org.bson.types.ObjectId;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+class MongoSplitPlanTest {
+ @Test
+ void integerBoundariesPreserveSignedWidthsAndUnboundedEnds() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt32(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 4, 4);
+ Assertions.assertEquals(4, plan.size());
+ var first = plan.partition(0).getArray("$and");
+ Assertions.assertEquals("int",
first.get(0).asDocument().getDocument("$expr")
+
.getArray("$in").get(1).asArray().get(0).asString().getValue());
+
Assertions.assertFalse(first.get(1).asDocument().getDocument("sequence").containsKey("$gte"));
+
Assertions.assertFalse(plan.partition(2).getArray("$and").get(1).asDocument()
+ .getDocument("sequence").containsKey("$lt"));
+
Assertions.assertTrue(plan.partition(3).getDocument("$expr").containsKey("$not"));
+ for (long value : new long[] {Long.MIN_VALUE, -2, -1, 0,
9007199254740992L,
+ 9007199254740993L, 9007199254740994L, Long.MAX_VALUE}) {
+ Assertions.assertEquals(1, matches(plan, new BsonInt64(value)),
Long.toString(value));
+ }
+ }
+
+ @Test
+ void duplicateQuantilesKeepFrequencyButNeverPublishDuplicateBoundaries() {
+ var samples = new ArrayList<BsonValue>();
+ for (int i = 0; i < 100; i++) {
+ samples.add(new BsonInt32(i < 80 ? 0 : i));
+ }
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 16,
8);
+ Assertions.assertTrue(plan.size() < 8);
+ for (long value = -1; value <= 101; value++) {
+ Assertions.assertEquals(1, matches(plan, new BsonInt64(value)));
+ }
+ samples.clear();
+ Assertions.assertEquals(1, matches(plan, new BsonInt32(81)));
+ }
+
+ @Test
+ void dateMillisecondsAreIndependentOfSqlPrecisionAndTimezone() {
+ var samples = List.<BsonValue>of(new BsonDateTime(Long.MIN_VALUE), new
BsonDateTime(-1),
+ new BsonDateTime(0), new BsonDateTime(1710064800123L), new
BsonDateTime(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("observed", "date", samples, 5, 5);
+ for (long value : new long[] {Long.MIN_VALUE, -2, -1, 0, 1,
1710064800122L,
+ 1710064800123L, 1710064800124L, Long.MAX_VALUE}) {
+ Assertions.assertEquals(1, matches(plan, new BsonDateTime(value)));
+ }
+ }
+
+ @Test
+ void objectIdsUseAllUnsignedBytes() {
+ var values = List.<BsonValue>of(new BsonObjectId(new
ObjectId("000000000000000000000000")),
+ new BsonObjectId(new ObjectId("7fffffffffffffffffffffff")),
+ new BsonObjectId(new ObjectId("800000000000000000000000")),
+ new BsonObjectId(new ObjectId("ffffffffffffffffffffffff")));
+ var plan = MongoSplitPlan.ranges("_id", "objectId", values, 4, 4);
+ for (BsonValue value : values) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+
+ @Test
+ void complementTestsTheValueItselfInsteadOfArrayElements() {
+ var plan = MongoSplitPlan.ranges("sequence", "integer", List.of(new
BsonInt32(0),
+ new BsonInt64(10), new BsonInt32(20)), 4, 4);
+ var expression =
plan.partition(0).getArray("$and").get(0).asDocument().getDocument("$expr");
+ Assertions.assertEquals(new org.bson.BsonString("$sequence"),
+ expression.getArray("$in").get(0).asDocument().get("$type"));
+ // Missing, null, arrays (even with integer members), double and
Decimal128 are NOT E.
+ for (BsonValue value : List.of(org.bson.BsonNull.VALUE, new
BsonArray(List.of(new BsonInt32(5))),
+ new org.bson.BsonDouble(5), new org.bson.BsonDecimal128(new
org.bson.types.Decimal128(
+ new
java.math.BigDecimal("5.000000000000000000000000000000001"))))) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+
+ @Test
+ void randomizedDuplicateAndSkewedSamplesCoverTheWholeIntegerDomain() {
+ var random = new Random(934351L);
+ for (int trial = 0; trial < 200; trial++) {
+ var values = new ArrayList<BsonValue>();
+ for (int i = 0; i < 1 + random.nextInt(150); i++) {
+ values.add(new BsonInt64(random.nextInt(7) == 0 ?
random.nextLong() : random.nextInt(5)));
+ }
+ int admitted = 2 + random.nextInt(31);
+ var plan = MongoSplitPlan.ranges("sequence", "integer", values,
32, admitted);
+ Assertions.assertTrue(plan.size() <= admitted);
+ for (int i = 0; i < 100; i++) {
+ Assertions.assertEquals(1, matches(plan, new
BsonInt64(random.nextLong())));
+ }
+ }
+ }
+
+ @Test
+ void oneRequestedSplitAndEmptySamplesNeverMeanEmptyResults() {
+ Assertions.assertEquals(new BsonDocument(),
MongoSplitPlan.ranges("sequence", "integer",
+ List.of(new BsonInt32(3)), 1, 1).partition(0));
+ Assertions.assertEquals(new BsonDocument(),
MongoSplitPlan.ranges("sequence", "integer",
+ List.of(), 8, 8).partition(0));
+ }
+
+ @Test
+ void partitionsAreOwnedAndUnsupportedFamiliesFail() {
+ var plan = MongoSplitPlan.ranges("sequence", "integer",
+ List.of(new BsonInt32(1), new BsonInt32(2), new BsonInt32(3)),
3, 3);
+ plan.partition(0).clear();
+ Assertions.assertFalse(plan.partition(0).isEmpty());
+ for (String family : List.of("decimal", "double", "string", "array",
"timestamp")) {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> MongoSplitPlan.ranges("sequence", family,
List.of(new BsonInt32(1)), 4, 4));
+ }
+ }
+
+ @Test
+ void idRangesUseTotalOrderExpressionsInsteadOfAComplement() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("_id", "integer", samples, 4, 4);
+ Assertions.assertEquals(4, plan.size());
+ for (int i = 0; i < plan.size(); i++) {
+ var expression = plan.partition(i).get("$expr");
+ Assertions.assertNotNull(expression, "Use expression comparisons
across BSON types");
+
Assertions.assertFalse(expression.asDocument().containsKey("$not"));
+
Assertions.assertFalse(plan.partition(i).toJson().contains("$type"));
+ }
+ Assertions.assertEquals(new BsonInt64(9007199254740993L),
plan.partition(1)
+
.getDocument("$expr").getArray("$and").get(1).asDocument().getArray("$lt").get(1));
+ }
+
+ @Test
+ void oneIdBoundaryValueUsesACompleteSingleScan() {
+ var plan = MongoSplitPlan.ranges("_id", "integer",
+ List.of(new BsonInt32(7), new BsonInt64(7)), 16, 16);
+ Assertions.assertEquals(1, plan.size());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+
+ @Test
+ void idRangesPreserveDateMillisecondsAndSignedIntegerEndpoints() {
+ for (String family : List.of("date", "integer")) {
+ var values = new ArrayList<BsonValue>();
+ for (long value : new long[] {Long.MIN_VALUE, -1, 0,
9007199254740993L, Long.MAX_VALUE}) {
+ values.add(family.equals("date") ? new BsonDateTime(value) :
new BsonInt64(value));
+ }
+ for (int admitted : new int[] {2, 3, 8, 32, 256}) {
+ var plan = MongoSplitPlan.ranges("_id", family, values, 256,
admitted);
+ Assertions.assertTrue(plan.size() <= admitted);
+ for (BsonValue value : values) {
+ Assertions.assertEquals(1, matches(plan, value));
+ }
+ }
+ }
+ }
+
+ @Test
+ void customFieldWithOnlyOneAdmittedRangeUsesACompleteSingleScan() {
+ var samples = List.<BsonValue>of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(-1),
+ new BsonInt64(9007199254740993L), new
BsonInt64(Long.MAX_VALUE));
+ var plan = MongoSplitPlan.ranges("sequence", "integer", samples, 8, 2);
+ Assertions.assertEquals(1, plan.size());
+ Assertions.assertEquals("insufficient_boundaries", plan.reason());
+ Assertions.assertEquals("", plan.family());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+
+ @Test
+ void identicalCustomFieldSamplesNeverPublishTwoUnboundedTypeScans() {
+ for (BsonValue value : List.of(new BsonInt64(9007199254740993L),
+ new BsonDateTime(Long.MIN_VALUE), new BsonObjectId(new
ObjectId("800000000000000000000001")))) {
+ String family = value.isInt64() ? "integer" : value.isDateTime() ?
"date" : "objectId";
+ for (int admitted : new int[] {2, 3, 8, 256}) {
+ var plan = MongoSplitPlan.ranges("sequence", family,
List.of(value, value, value), 256, admitted);
+ Assertions.assertEquals(1, plan.size(), family + ":" +
admitted);
+ Assertions.assertEquals("insufficient_boundaries",
plan.reason());
+ Assertions.assertEquals(new BsonDocument(), plan.partition(0));
+ }
+ }
+ }
+
+ @Test
+ void encodedPartitionsAreExactAndOwnedByEachDescriptor() {
+ for (String family : List.of("integer", "date", "objectId")) {
+ List<BsonValue> values = switch (family) {
+ case "integer" -> List.of(new BsonInt64(Long.MIN_VALUE), new
BsonInt64(9007199254740993L),
+ new BsonInt64(Long.MAX_VALUE));
+ case "date" -> List.of(new BsonDateTime(Long.MIN_VALUE), new
BsonDateTime(-1),
+ new BsonDateTime(Long.MAX_VALUE));
+ default -> List.of(new BsonObjectId(new
ObjectId("000000000000000000000001")),
+ new BsonObjectId(new
ObjectId("800000000000000000000001")),
+ new BsonObjectId(new
ObjectId("ffffffffffffffffffffffff")));
+ };
+ var plan = MongoSplitPlan.ranges("_id", family, values, 3, 3);
+ var budgeted = plan.withDescriptorBudget(1024);
+ for (int index = 0; index < plan.size(); index++) {
+ byte[] expected = MongoSplitPlan.encode(plan.partition(index));
+ byte[] exposed = budgeted.partitionBytes(index);
+ Assertions.assertArrayEquals(expected, exposed);
+ java.util.Arrays.fill(exposed, (byte) 0);
+ Assertions.assertArrayEquals(expected,
plan.partitionBytes(index));
+ Assertions.assertArrayEquals(expected,
budgeted.partitionBytes(index));
+ Assertions.assertArrayEquals(expected,
MongoSplitPlan.encode(plan.partition(index)));
+ }
+ }
+ }
+
+ private static int matches(MongoSplitPlan plan, BsonValue value) {
Review Comment:
The oracle infers `scalar` from the Java value class and never reads the
`$type` list production emits (`$in: [{$type: {$field}}, types]`), and it
identifies the complement from that same inferred class. Emitting `["int"]`
instead of `["int","long"]`, or a wrong `date`/`objectId` type name, is
therefore invisible to every `matches() == 1` assertion; the only guard on the
`$type` contract is the single `"int"` check in
`integerBoundariesPreserveSignedWidthsAndUnboundedEnds`.
Since this is the only coverage/disjointness oracle, please evaluate the
emitted `$type` list against the value's BSON type name, and assert the full
list per family and across partitions.
##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoRangePlanner.java:
##########
@@ -0,0 +1,370 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.mongodb;
+
+import org.apache.doris.connector.spi.ConnectorSession;
+import org.apache.doris.connector.spi.ConnectorStatementScope;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ConnectorScanContext;
+import org.apache.doris.thrift.TMongoSnapshot;
+
+import org.bson.BsonValue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/** Complete planning precedes publication. Statement memos retain either that
plan or its failure. */
+final class MongoRangePlanner {
+ static final int SAMPLE_ROWS = 4096;
+ static final int PROBE_TIMEOUT_MS = 2000;
+ private final String memoKey = "mongodb.split.plan:" + UUID.randomUUID();
+ private final Supplier<MongoCatalogClient> client;
+
+ MongoRangePlanner(Supplier<MongoCatalogClient> client) {
+ this.client = client;
+ }
+
+ MongoSplitPlan plan(ConnectorSession session, MongoTableHandle table,
TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, long
descriptorBytes) {
+ checkCancelled(session, context);
+ var options = session.getSessionProperties();
+ String strategy = options.getOrDefault("mongodb_split_strategy",
"single");
+ if (!Set.of("single", "auto", "range").contains(strategy)) {
+ throw new IllegalArgumentException("Invalid
mongodb_split_strategy");
+ }
+ if ("single".equals(strategy)) {
+ return MongoSplitPlan.single(1, 1, "single_strategy");
+ }
+ String field = options.getOrDefault("mongodb_split_field", "");
+ boolean custom = !field.isEmpty();
+ if (!custom) {
+ field = "_id";
+ }
+ if (!MongoPredicateTranslator.safePath(field) || field.isEmpty() ||
field.length() > 256) {
+ throw new IllegalArgumentException("MongoDB split field must be an
unambiguous top-level field");
+ }
+ String selectedField = field;
+ Budget budget =
session.getStatementScope().computeIfAbsent("mongodb.split.budget",
Budget::new);
+ String tableKey = tableKey(session, table);
+ // Count fallback scans too: a view must not make a global custom
field look unambiguous.
+ if (custom) {
+ budget.customField(tableKey);
+ }
+ Memo memo = session.getStatementScope().computeIfAbsent(tableKey,
Memo::new);
+ return memo.get(session, context, () -> load(session, table, snapshot,
preference, context,
+ strategy, selectedField, descriptorBytes, budget));
+ }
+
+ Optional<MongoSplitPlan> planned(ConnectorSession session,
MongoTableHandle table) {
+ Memo memo = session.getStatementScope().getIfPresent(tableKey(session,
table));
+ return memo == null ? Optional.empty() : memo.completed();
+ }
+
+ private String tableKey(ConnectorSession session, MongoTableHandle table) {
+ // One provider can see several namespaces; never reuse another
collection's bounds/admission.
+ return memoKey + ":" + session.getCatalogId() + ":" +
table.getSchema().getVersion();
+ }
+
+ private MongoSplitPlan load(ConnectorSession session, MongoTableHandle
table, TMongoSnapshot snapshot,
+ String preference, ConnectorScanContext context, String strategy,
String field,
+ long descriptorBytes, Budget budget) {
+ int maximum = splitLimit(session);
+ targetBytes(session, -1);
+ int requested = Math.min(8, maximum);
+ int descriptorMaximum = (int) Math.min(maximum, (8L * 1024 * 1024) /
descriptorBytes);
+ if (descriptorMaximum == 0) {
+ throw new DorisConnectorException("MongoDB split descriptor
exceeds planning byte budget");
+ }
+ String source = snapshot == null ? "catalog:" + session.getCatalogId()
+ : snapshot.getSourceKind() + ":" + snapshot.getSourceId();
+ if (!"collection".equals(table.getObjectKind())) {
+ return fallback(strategy, requested, "unsupported_object", source,
descriptorBytes, budget);
+ }
+ if (snapshot == null || session.getStatementScope() ==
ConnectorStatementScope.NONE) {
+ return fallback(strategy, requested, "snapshot_required", source,
descriptorBytes, budget);
+ }
+ if (descriptorMaximum == 1) {
+ return single(requested, maximum == 1 ? "split_limit" :
"descriptor_budget",
+ source, descriptorBytes, budget);
+ }
+ long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(
+ Math.min(PROBE_TIMEOUT_MS,
MongoSnapshot.remainingMillis(session)));
+ MongoSplitProbe probe;
+ try {
+ budget.acquireProbe(session, context, deadline);
+ try {
+ checkCancelled(session, context);
+ probe = client.get().probeSplits(table, field, preference,
snapshot, SAMPLE_ROWS,
+ remaining(session, context, deadline), () ->
checkCancelled(session, context));
+ budget.sampled(probe.sampleBytes());
+ } finally {
+ budget.probes.release();
+ }
+ } catch (PlanningLimit limit) {
+ checkCancelled(session, context);
+ MongoSnapshot.remainingMillis(session);
+ // Only optional pre-publication tuning can fall back.
Cancellation, authorization,
+ // history and source errors must never be converted into a
successful full scan.
+ return fallback(strategy, requested, "probe_budget", source,
descriptorBytes, budget);
+ }
+ checkCancelled(session, context);
+ MongoSnapshot.remainingMillis(session);
+ long targetBytes = targetBytes(session, probe.logicalBytes());
+ long estimate = probe.logicalBytes() < 0 ? 8
+ : Math.max(1, probe.logicalBytes() / targetBytes +
(probe.logicalBytes() % targetBytes == 0 ? 0 : 1));
+ requested = (int) Math.min(maximum, estimate);
+ if (!probe.indexed()) {
+ return fallback(strategy, requested, "missing_index", source,
descriptorBytes, budget);
+ }
+ String family = family(probe.samples());
+ if (family.isEmpty()) {
+ return probe.samples().isEmpty()
+ ? single(requested, "empty_sample", source,
descriptorBytes, budget)
+ : fallback(strategy, requested, "unsupported_type",
source, descriptorBytes, budget);
+ }
+ if ("auto".equals(strategy) && probe.logicalBytes() < 0) {
+ return single(requested, "unknown_complement_cost", source,
descriptorBytes, budget);
+ }
+ // Division avoids overflow when a valid shared split-size setting is
near Long.MAX_VALUE.
+ if ("auto".equals(strategy) && probe.logicalBytes() / targetBytes < 2)
{
Review Comment:
For a custom `mongodb_split_field`, every range plan still carries one
complement partition (`$expr: {$not: [$in: [{$type: "$field"}, [types]]]}` in
`MongoSplitPlan`), which cannot use an index and scans the whole collection.
Total documents examined is therefore ~2N and the wall-clock gain over one
cursor is bounded by roughly `splitCount / 2`, but this `auto` gate only looks
at `logicalBytes / targetBytes`. Please either raise the threshold for custom
fields or state the ~2x read amplification next to the `auto` documentation, so
opting into `auto` does not silently double source I/O for a fraction of the
parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]