morningman opened a new pull request, #70664:
URL: https://github.com/apache/doris/pull/70664

   ### Background
   
   Doris plans external-file byte ranges in FE and assigns them to BE scanners. 
Native Parquet and ORC readers then select the row groups or stripes belonging 
to each range. FE range sizing determines the initial work distribution, while 
the physical file layout determines what a reader can scan independently.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #66802 (branch-4.2 source), #68143 (Iceberg missing-file policy 
in split planning)
   PR Type: Enhancement
   
   Problem Summary:
   
   This ports the two-level external-file splitting from branch-4.2 #66802 to 
the current scanner and connector architecture. FE selects coarse ranges for 
eligible native scans, and BE refines them into row-group or stripe tasks that 
scanners in the same instance can consume. The port also handles source 
completion, shared immutable metadata, runtime filters, condition-cache 
publication, and the plugin request paths. Eligible scans use dedicated FE and 
BE size targets after upgrading.
   
   Original author: @Gabriel39
   
   ### Details
   
   - **Change map:** 2,247 changed production lines across 49 files; 2,562 test 
lines across 25 files; two build-metadata lines in one file; no generated 
outputs or documentation. BE adds physical split scheduling, exact ownership 
and parent-scoped metadata/cache state. FE and SPI propagate coarse sizing 
through synchronous, partition-batch and streaming planning; connectors retain 
their source-specific exclusions. Tests port the original BE cases and cover 
the new engine/plugin seams.
   - **Before and after:** FE previously used each source's existing split 
policy. Eligible Parquet/ORC scans now use a 512 MiB FE target; BE groups 
physical children toward 64 MiB. For a 96 MiB Iceberg fixture without explicit 
split offsets, tests verify three ranges with a 32 MiB coarse hint and two with 
64 MiB, including streaming. Files with Iceberg deletes, logical/JNI Paimon 
splits, metadata COUNT and serial scans retain their applicable fallback. BE 
may refine eligible ranges even when FE omits the target; a positive target 
controls coalescing only.
   - **Configs and defaults:** Add session variables 
`file_split_size_on_fe=536870912` and `file_split_size_on_be=67108864`, both 
forwarded and changeable through SET. Nonpositive values keep legacy FE 
planning; they do not disable BE refinement. The deprecated 
`enable_file_scanner_v2` compatibility variable remains ineffective.
   - **Impact on existing users:** Eligible ranges use the dedicated targets 
instead of `file_split_size`; that variable remains the fallback policy. 
Results and table-format delete semantics are preserved. Load retains FE range 
boundaries. The optional Thrift field uses the same id 17 as 4.1/4.2. Older BE 
ignores it and cannot recover coarse-range parallelism, so upgrade BE first. 
Connector API becomes 33.0; plugins must be rebuilt together with FE.
   
   ### Testing
   
   - Standard FE test script: 790 tests in 52 classes passed after rebasing 
onto current master, including the 533 affected engine/connector tests, the 
complete 163-test SPI suite and 94 predicate/converter tests covering the newly 
merged changes. Both recorded SPI surface contracts passed.
   - Five targeted temporary mutants lost either synchronous/batch request 
propagation or one connector's reuse-key input. Every control passed and every 
mutant failed its intended assertion.
   - Before the final rebase, all 27 modified BE translation units passed 
syntax checks against the generated compilation database; a deliberate failing 
probe confirmed detection. BE and protocol sources are unchanged since.
   - Five changed regression scripts passed Groovy compilation before the final 
rebase and are unchanged since. Actual BE unit tests and external end-to-end 
suites have not run: the installed third-party package lacks the BSON/MongoDB 
2.0 include directories required to generate the BE build. No running-cluster 
evidence is claimed.
   
   ### Notes for Reviewers
   
   Start with source-split publication and completion, then physical ownership 
and cache finalization. Children share only their parent's immutable metadata 
and synchronized hints; decoding, predicates and progress remain reader-local. 
Keep the refinement-planning missing-file classifier aligned with normal 
Iceberg reads.
   
   Iceberg preserves SDK split offsets, so an explicit offset list can override 
the requested size. Existing tests that deliberately force tiny FE ranges now 
disable coarse FE planning to retain their coverage. The new regression 
compares parallel configurations with a single-scanner reference and requires 
no handwritten expected output.
   
   ### Release note
   
   Native Parquet and ORC external scans gain FE coarse splitting and BE 
physical refinement, controlled by `file_split_size_on_fe` (512 MiB) and 
`file_split_size_on_be` (64 MiB). Eligible scans use these targets instead of 
`file_split_size`. Set a nonpositive dedicated size to retain legacy FE sizing; 
BE can still refine eligible ranges without a coalescing target. Upgrade BE 
before FE and rebuild connector plugins for API 33.0.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; update the external scan split policy 
and both session-variable entries in apache/doris-website development 
documentation.
   - Pre-submission status: Partially verified: BE unit execution and external 
end-to-end regression are blocked by missing third-party BSON/MongoDB 2.0 
dependencies.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Port adaptations, tests, evidence collection, and this 
description.
   - Human review: not stated
   - Prompts: Start the next item from the batch handoff. Follow the settled 
decision to retain branch-4.2 BE refinement without an FE authorization gate, 
and use engine-coordinated FE/SPI planning in one PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to