github-actions[bot] commented on code in PR #70708:
URL: https://github.com/apache/doris/pull/70708#discussion_r4640897060


##########
be/src/format_v2/table/mongodb_reader.cpp:
##########
@@ -220,16 +261,29 @@ class NativeMongoStream final : public MongoCursorStream {
 public:
     ~NativeMongoStream() override { close(); }
     Status open(const TMongoScanDesc& d, std::function<bool()> cancelled) 
override {
-        _cancelled = std::move(cancelled);
-        _deadline = Clock::now() + std::chrono::milliseconds(d.timeout_ms);
+        if (_client) {
+            return Status::InvalidArgument("MongoDB stream requires close 
before reopening");
+        }
+        if (!_pool) {
+            _cancelled = std::move(cancelled);
+            _deadline = Clock::now() + std::chrono::milliseconds(d.timeout_ms);
+            RETURN_IF_ERROR(check_deadline());
+            RETURN_IF_ERROR(validate_pushdown_bson(d));
+            RETURN_IF_ERROR(validate_snapshot_descriptor(d));
+            RETURN_IF_ERROR(acquire_pool(d, &_pool));
+        }
+        // Preserve the first deadline across cooperative retries. Sleeping 
here can occupy a
+        // workload group's only scanner worker while the client holder is 
queued to resume.
         RETURN_IF_ERROR(check_deadline());
-        RETURN_IF_ERROR(validate_pushdown_bson(d));
-        RETURN_IF_ERROR(validate_snapshot_descriptor(d));
-        RETURN_IF_ERROR(acquire_pool(d, &_pool));
-        while (!(_client = mongoc_client_pool_try_pop(_pool->pool.get()))) {
+        _client = mongoc_client_pool_try_pop(_pool->pool.get());

Review Comment:
   [P1] Prevent pool exhaustion by a dependency-blocked probe scan
   
   Cooperative checkout still permits a cross-context deadlock. On one BE, 
hash-join two large collections in the same catalog with 
`mongodb.pool_max_clients=2`, snapshot/range, `file_split_size=1`, 
`max_file_split_num=2`, two file scanners, one pipeline task and runtime 
filters disabled. `PipelineTask::_open()` starts each scan context before 
checking the join's build dependency, so both probe readers can acquire the 
clients and publish non-EOF batches first. Their consumer then waits for the 
build; those readers retain their leases until resumed/closed, while every 
build reader only retries here. Per-context deque fairness cannot break this 
cycle, and the query times out. The same two single-cursor scans fit the pool. 
Please make admission account for blocked consumers/dependencies, or otherwise 
ensure their cursors cannot retain all clients required by the build. Add a 
same-catalog join with multi-batch partitions; the existing 601-row join oracle 
releases leases within one batch.
  This differs from the addressed same-context worker/admission starvation.



##########
fe/fe-connector/fe-connector-mongodb/src/main/java/org/apache/doris/connector/mongodb/MongoDriverCatalogClient.java:
##########
@@ -177,6 +183,131 @@ public BsonTimestamp selectSnapshot(String database, 
String collection, String p
         }
     }
 
+    @Override
+    public MongoSplitProbe probeSplits(MongoTableHandle table, String field, 
String preference,
+            TMongoSnapshot snapshot, int rows, long timeoutMs, Runnable 
checkCancelled) {
+        long deadline = System.nanoTime() + 
TimeUnit.MILLISECONDS.toNanos(timeoutMs);
+        String index = null;
+        long bytes = 0;
+        try {
+            var collection = probeDatabase(table.getDatabase(), deadline, 
checkCancelled)
+                    .getCollection(table.getName());
+            try (MongoCursor<Document> indexes = 
collection.listIndexes().batchSize(1)
+                    .timeoutMode(TimeoutMode.CURSOR_LIFETIME).iterator()) {
+                int count = 0;
+                while (true) {
+                    // hasNext may issue getMore. Cancellation must be checked 
before that request,
+                    // not only after it returns; the lifetime timeout also 
covers all cursor batches.
+                    checkProbe(deadline, checkCancelled);
+                    if (!indexes.hasNext()) {
+                        break;
+                    }
+                    checkProbe(deadline, checkCancelled);
+                    Document value = indexes.next();
+                    bytes += 
MongoSplitPlan.encode(value.toBsonDocument()).length;
+                    if (++count > 128 || bytes > 65536) {
+                        throw new MongoRangePlanner.PlanningLimit();
+                    }
+                    if (eligibleIndex(value, field)) {

Review Comment:
   [P2] Reject compound text indexes before admitting range scans
   
   A legal `{split_key:1, body:"text"}` index is reported as `{split_key:1, 
_fts:"text", _ftsx:1}` and need not have `sparse:true`, so `eligibleIndex` 
accepts it and enumeration stops here. With no ordinary `split_key` index, 
diverse integer keys, snapshot `auto`/`range`, `file_split_size=1` and 
`max_file_split_num=8`, this admits seven typed ranges plus the complement. By 
static tracing, each becomes a full collection scan: [MongoDB's planner rejects 
inequalities on TEXT-index 
prefixes](https://github.com/mongodb/mongo/blob/r7.0.28/src/mongo/db/query/planner_ixselect.cpp#L556-L586),
 and these predicates have no `$text`. Source work grows to about 8N instead of 
a single N-document scan. Please check component types across the complete key 
pattern and skip unsupported index families, retaining `auto` fallback / 
explicit-range rejection when no B-tree qualifies. This is separate from the 
fixed wildcard/hinted-sample error: the unhinted data partitions themselves 
cannot use this TEXT pr
 efix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to