suxiaogang223 commented on code in PR #66521:
URL: https://github.com/apache/doris/pull/66521#discussion_r3765102752
##########
fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeScanProfileTest.java:
##########
@@ -87,4 +90,22 @@ public void sharesGroupAcrossScans() {
Assertions.assertEquals("3", group.getChildMap().get("Table Scan
(db.a)").getInfoString("data_files"));
Assertions.assertEquals("5", group.getChildMap().get("Table Scan
(db.b)").getInfoString("data_files"));
}
+
+ @Test
+ public void concurrentStreamingScansShareOneGroup() {
+ RuntimeProfile summary = new RuntimeProfile("Execution Summary");
+ List<CompletableFuture<Void>> writes = new ArrayList<>();
+ for (int i = 0; i < 32; i++) {
+ String label = "Table Scan (db.t" + i + ")";
Review Comment:
Fixed in daaae7c1f91. The probabilistic common-pool test is replaced by a
two-thread latch-controlled RuntimeProfile harness that deterministically
forces the old null-lookup/add race and passes with synchronized group creation.
##########
fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java:
##########
@@ -282,12 +282,13 @@ default OptionalLong
scannedPartitionCount(List<ConnectorScanRange> scanRanges)
* {@link ConnectorScanProfile} groups the engine writes into the query's
profile execution summary.
*
* <p>The default returns an empty list (connector reports nothing). A
connector that wants scan
- * diagnostics harvests them from its SDK during {@code planScan} (the
paimon SDK exposes a metric
- * registry, the iceberg SDK a metrics reporter), stashes them keyed by
{@link ConnectorSession#getQueryId()},
- * and drains them here — mirroring the per-query queryId stashes this SPI
already uses (read-transaction
- * release, rewritable-delete supply). The engine calls this immediately
after {@code planScan} on the
- * same thread, so the harvest is complete; the connector must also drop
its stash on
- * {@link #releaseReadTransaction} to reclaim any entry a thrown {@code
planScan} left behind.</p>
+ * diagnostics harvests them from its SDK during {@code planScan} or
streaming split generation (the paimon
+ * SDK exposes a metric registry, the iceberg SDK a metrics reporter),
stashes them keyed by
+ * {@link ConnectorSession#getQueryId()}, and drains them here — mirroring
the per-query queryId stashes this
+ * SPI already uses (read-transaction release, rewritable-delete supply).
The engine calls this immediately
+ * after {@code planScan}, or after closing a streaming split source, so
the harvest is complete; the connector
Review Comment:
Fixed in daaae7c1f91. Streaming now uses a separate default-empty
`collectStreamingScanProfiles` opt-in hook instead of invoking the eager
collector. Iceberg explicitly opts in. Per the repository surface policy, the
connector API is bumped from 4.0 to 5.0 and the frozen surface baseline is
refreshed; targeted SPI tests and the full FE build pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
