924060929 opened a new pull request, #69683: URL: https://github.com/apache/doris/pull/69683
### What problem does this PR solve? Issue Number: None Problem Summary: External scans currently enter Batch Mode only above 1024 partitions or files. Even in Batch Mode, a producer batch defaults to 128 splits and each backend queue is bounded by 32 **batches**. That couples the memory bound to batch size and delays small LIMIT scans while large groups are prepared. This PR enables Batch Mode from one partition or file by default. It separates the Hive partition-processing batch size (`split_partition_batch_size`, default 16) from the Batch Mode trigger. Producers still assign and offer splits in batches, but each backend queue stores individual splits and enforces a capacity of 1024 splits; a partly full queue accepts only the available slots and wakes the producer when the consumer drains it. The producer batch defaults to 8 splits. The consumer still obeys the BE request batch size, while preserving cancellation, first-fetch limits, backpressure, and error propagation. The new connector SPI method advances the plugin API version to 25 and updates its recorded surface. In a matched, one-BE Release benchmark on the integrated test branch, refreshed-metadata LIMIT medians at `split_batch_size=8` versus `32` were **75.8 vs 102.5 ms** for Hive 10k partitions and **23.5 vs 30.6 ms** for Iceberg 1000 manifests. Eight beat 32 in 11/12 paired blocks in each LIMIT scenario. Warm full scans had no stable winner among 8/16/32/64. These numbers select the producer batch size; they are not a fresh end-to-end benchmark of this rebased PR against current master. ### Release note External table Batch Mode now starts at one partition or file by default. The default producer batch size is 8 splits, queue capacity is 1024 splits per backend, and `split_partition_batch_size` controls Hive partition-processing batches separately. ### Check List (For Author) - Test: Unit Test. Before rebasing onto current master, FE build and 10 focused FE test classes passed (162 tests). On current master, `ConnectorPluginSurfaceTest` (2), `ConnectorMetadataSurfaceTest` (4), and `HiveScanBatchModeTest` (36) passed with Checkstyle. The current FE-core Maven test attempt could not resolve local snapshot artifacts without `-am`; the full reactor path requires an x86 gRPC generator unavailable on this ARM host. CI build is requested below. - Behavior changed: Yes, Batch Mode trigger and split-buffering defaults change as described above. - Does this need documentation: No separate doc PR; the connector scheduling document is updated here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
