morningman opened a new pull request, #70664: URL: https://github.com/apache/doris/pull/70664
### Background Doris plans external-file byte ranges in FE and assigns them to BE scanners. Native Parquet and ORC readers then select the row groups or stripes belonging to each range. FE range sizing determines the initial work distribution, while the physical file layout determines what a reader can scan independently. ### What problem does this PR solve? Issue Number: None Related PR: #66802 (branch-4.2 source), #68143 (Iceberg missing-file policy in split planning) PR Type: Enhancement Problem Summary: This ports the two-level external-file splitting from branch-4.2 #66802 to the current scanner and connector architecture. FE selects coarse ranges for eligible native scans, and BE refines them into row-group or stripe tasks that scanners in the same instance can consume. The port also handles source completion, shared immutable metadata, runtime filters, condition-cache publication, and the plugin request paths. Eligible scans use dedicated FE and BE size targets after upgrading. Original author: @Gabriel39 ### Details - **Change map:** 2,247 changed production lines across 49 files; 2,562 test lines across 25 files; two build-metadata lines in one file; no generated outputs or documentation. BE adds physical split scheduling, exact ownership and parent-scoped metadata/cache state. FE and SPI propagate coarse sizing through synchronous, partition-batch and streaming planning; connectors retain their source-specific exclusions. Tests port the original BE cases and cover the new engine/plugin seams. - **Before and after:** FE previously used each source's existing split policy. Eligible Parquet/ORC scans now use a 512 MiB FE target; BE groups physical children toward 64 MiB. For a 96 MiB Iceberg fixture without explicit split offsets, tests verify three ranges with a 32 MiB coarse hint and two with 64 MiB, including streaming. Files with Iceberg deletes, logical/JNI Paimon splits, metadata COUNT and serial scans retain their applicable fallback. BE may refine eligible ranges even when FE omits the target; a positive target controls coalescing only. - **Configs and defaults:** Add session variables `file_split_size_on_fe=536870912` and `file_split_size_on_be=67108864`, both forwarded and changeable through SET. Nonpositive values keep legacy FE planning; they do not disable BE refinement. The deprecated `enable_file_scanner_v2` compatibility variable remains ineffective. - **Impact on existing users:** Eligible ranges use the dedicated targets instead of `file_split_size`; that variable remains the fallback policy. Results and table-format delete semantics are preserved. Load retains FE range boundaries. The optional Thrift field uses the same id 17 as 4.1/4.2. Older BE ignores it and cannot recover coarse-range parallelism, so upgrade BE first. Connector API becomes 33.0; plugins must be rebuilt together with FE. ### Testing - Standard FE test script: 790 tests in 52 classes passed after rebasing onto current master, including the 533 affected engine/connector tests, the complete 163-test SPI suite and 94 predicate/converter tests covering the newly merged changes. Both recorded SPI surface contracts passed. - Five targeted temporary mutants lost either synchronous/batch request propagation or one connector's reuse-key input. Every control passed and every mutant failed its intended assertion. - Before the final rebase, all 27 modified BE translation units passed syntax checks against the generated compilation database; a deliberate failing probe confirmed detection. BE and protocol sources are unchanged since. - Five changed regression scripts passed Groovy compilation before the final rebase and are unchanged since. Actual BE unit tests and external end-to-end suites have not run: the installed third-party package lacks the BSON/MongoDB 2.0 include directories required to generate the BE build. No running-cluster evidence is claimed. ### Notes for Reviewers Start with source-split publication and completion, then physical ownership and cache finalization. Children share only their parent's immutable metadata and synchronized hints; decoding, predicates and progress remain reader-local. Keep the refinement-planning missing-file classifier aligned with normal Iceberg reads. Iceberg preserves SDK split offsets, so an explicit offset list can override the requested size. Existing tests that deliberately force tiny FE ranges now disable coarse FE planning to retain their coverage. The new regression compares parallel configurations with a single-scanner reference and requires no handwritten expected output. ### Release note Native Parquet and ORC external scans gain FE coarse splitting and BE physical refinement, controlled by `file_split_size_on_fe` (512 MiB) and `file_split_size_on_be` (64 MiB). Eligible scans use these targets instead of `file_split_size`. Set a nonpositive dedicated size to retain legacy FE sizing; BE can still refine eligible ranges without a coalescing target. Upgrade BE before FE and rebuild connector plugins for API 33.0. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; update the external scan split policy and both session-variable entries in apache/doris-website development documentation. - Pre-submission status: Partially verified: BE unit execution and external end-to-end regression are blocked by missing third-party BSON/MongoDB 2.0 dependencies. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Port adaptations, tests, evidence collection, and this description. - Human review: not stated - Prompts: Start the next item from the batch handoff. Follow the settled decision to retain branch-4.2 BE refinement without an FE authorization gate, and use engine-coordinated FE/SPI planning in one PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
