suxiaogang223 opened a new pull request, #70806:
URL: https://github.com/apache/doris/pull/70806

   ### Background
   
   Doris reads safe Paimon raw files with its Native ORC/Parquet readers. Rust 
and JNI handle logical splits that require table-format semantics.
   
   ### What problem does this PR solve?
   
   Issue Number: #70797
   Related PR: None
   PR Type: Enhancement
   
   Problem Summary:
   
   Enabling Rust currently diverts compatible raw files from Native and 
conditionally skips Native historical timestamp checks. Complete Native 
eligibility first, then consider Rust and JNI. Refresh integer and collect 
eligibility assumptions against paimon-rust 0.5.0 while retaining independent 
resource gates.
   
   ### Details
   
   - **Change map:** 82 changed production lines adjust routing and aggregate 
compatibility; 225 changed test lines cover precedence, non-raw column merging, 
compatibility gates and regression profiles. No generated files or dependency 
pins change.
   - **Before and after:** Ordinary raw-convertible ORC/Parquet scans remain 
Native when Rust is enabled. Unsupported Native splits still consider 
compatible Rust, then JNI. Metadata counts and Native footer aggregates retain 
their fast paths.
   - **Configs and defaults:** No new configuration or default change. Rust 
remains opt-in, forcing JNI retains precedence, and diagnostic split 
suppression drops ranges without rerouting them.
   - **Impact on existing users:** Integer sum/product and non-distinct ARRAY 
collect become recognized by the compatibility helper. Independent gates still 
exclude unsupported primary-key merge resources, projections and storage 
configurations.
   - **Behavior change:** Reader preference becomes Native > compatible/enabled 
Rust > JNI after metadata COUNT. Native historical time/schema checks always 
complete before selection.
   
   ### Testing
   
   - `./build.sh --fe --be-java-extensions -j 2` and `./build.sh --be -j 4`: 
passed, including Checkstyle; BE used ASAN and the latest official thirdparty 
prebuilt.
   - Preset FE runner: 152 tests passed, zero failures/errors/skips. The 
Native-priority assertion fails on the original implementation. A real two-file 
column merge with overlapping row IDs selects Rust, then disabled/forced JNI.
   - Preset local regression passed script-generated output and verification: 
ordered results, 21 COMPLETE reader profiles, Native ORC/Parquet, Rust column 
merges, JNI parity, both deprecated scanner flag values, metadata count and 
footer/full-scan controls.
   - Independent prebuilt C ABI probes matched Java engine/format controls, 
integer overflow behavior and non-distinct collect. Distinct collect ordering 
differs and remains gated. These probes are separate from integrated Doris 
execution.
   - Regression framework and changed Groovy syntax checks passed. The two 
changed external HDFS/MinIO suites have not run.
   
   ### Notes for Reviewers
   
   The local cluster used an 8 GiB process limit and a 256 MiB memory file 
cache: Btrfs zero inode counts triggered an existing disk-cache monitor 
division by zero during initial startup. Disk-cache behavior remains unverified.
   
   SDK/C ABI patch-dependent functionality is deferred. Merge budgets, 
cancellation, recursive read types/Variant, FileIO and incremental integration 
remain tracked in #70797 and [paimon-rust 
#1468](https://github.com/apache/paimon-rust/issues/1468). This PR does not 
complete the full 0.5.0 integration or add runtime JNI retries.
   
   ### Release note
   
   Enabling the Paimon Rust reader preserves Native execution for compatible 
raw ORC/Parquet files, including footer aggregates.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (reader-selection documentation update 
tracked with #70797)
   - Pre-submission status: Partially verified: local reader results/profiles 
passed; changed external HDFS/MinIO suites have not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code, tests, analysis and this description.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to