suxiaogang223 opened a new issue, #70797:
URL: https://github.com/apache/doris/issues/70797

   ### Background
   
   Doris can read safe raw Paimon files with its Native ORC/Parquet readers and 
use a Paimon SDK for logical reads. The optional Rust reader currently precedes 
Native in compatible ordinary scans. Its compatibility gates also mix actual 
SDK restrictions, incomplete C ABI/host integration, and assumptions that 
predate paimon-rust 0.5.0.
   
   ### Expected behavior
   
   Preserve accurate metadata COUNT, then choose Native > Rust > JNI. 
`force_jni_scanner` still overrides data readers. Enabling Rust should prefer 
it where a Paimon logical reader is needed, while preserving Native correctness 
checks, footer aggregates, deletion vectors, diagnostic split suppression, and 
mixed-backend capability checks.
   
   ### Scope
   
   Align ordinary queries, including the incremental query forms already 
exposed by Doris, against paimon-rust v0.5.0 code 
`bb6389a8b430e897e038ff7395b6a7763d785a7a` and Java Paimon 2.1.0. Do not infer 
0.5.0 support from Rust main/0.6. Writes and dedicated full-text/vector/hybrid 
optimizer pushdown are separate work.
   
   ### Proposed implementation and dependencies
   
   1. Evaluate Native eligibility completely before considering Rust, including 
historical ORC and cross-family timestamp checks. Update the old priority 
assertions.
   2. Establish a query-shared resource and cancellation contract before 
lifting multi-file primary-key or partial-update/aggregation gates. Include 
retained merge inputs, same-key versions, aggregate state, pending Arrow 
output, concurrent splits, exhaustion, and cleanup. Small batches and 
`with_resources` alone are insufficient.
   3. Rebuild the function × type × option matrix against the fixed source. 
Integer sum/product wrap by default and collect/map/nested/sketch functions 
exist; retaining an outdated SDK explanation is incorrect. Keep actual 
unsupported Java SPI or semantic combinations on JNI.
   4. Adapt recursive projections and Variant carriers across C/Arrow/Doris. 
The core read-type API is not exposed by the current root-name C projection 
API; Rust's Variant child order is value/metadata while Doris's native Arrow 
carrier expects metadata/value. Do not remove gates without a correct mapping 
and mixed-BE negotiation.
   5. Verify schema evolution, timestamp range/timezones, DV/MOR, external 
paths, credentials/FileIO/cache, and existing incremental query semantics. 
Retain concrete fallback reasons for unintegrated contracts.
   
   SDK dependencies include apache/paimon-rust#1386 (pending-output batch 
retention), apache/paimon-rust#1468 (reader budget coverage and C host-pool 
integration), and apache/paimon-rust#1382 (audit/incremental C execution). A 
patched SDK needs a tested 0.5.0-compatible backport and a matching thirdparty 
prebuilt; changing to a newer SDK version is not equivalent to integrating the 
pinned version.
   
   ### Evidence so far
   
   On Doris master `5bc6ed62f9b01cd2cecaf9a3420d5ee015638ec1`, the Native-first 
candidate passes 148 FE tests: 128 scan-provider, 12 Rust-selector, 8 
capability tests. The new raw-file priority assertion fails against the 
original provider and passes against the candidate.
   
   Independent C ABI probes using the 2026-11-19 Doris prebuilt matched Java 
2.1.0 ordered results for four merge-engine controls in Parquet and ORC. These 
are SDK probes, not proof that Doris executed its Rust reader.
   
   Resource probes reproduced a gap: ORC reads succeed with a zero-byte 
resource context and report zero peak reservations. For two overlapping files 
with one early survivor and 32768 subsequent deletes, Parquet's reported peak 
reservation stays 44020 bytes under a 2 MiB budget, while process peak RSS 
changes from 27236 to 95996 KiB when read.batch-size changes from 1 to 32. RSS 
includes runtime/decoder overhead; the exact retained-state accounting still 
needs deterministic tests. Batch size 1 is a control, not a fix.
   
   ### Acceptance criteria
   
   - [ ] Native priority with COUNT/footer optimizations, complete Native time 
checks, force-JNI, ignore_split_type and mixed-BE coverage.
   - [ ] Real non-raw-convertible tasks execute Doris Rust, confirmed by 
completed Profiles and ordered JNI comparisons; raw append scans remain Native.
   - [ ] Four merge engines, updates/deletes, multiple files/levels, DV, 
sequence/aggregation options and invalid combinations have result/error 
coverage.
   - [ ] Shared memory budget, oversized same-key state, sparse output, 
concurrency, cancellation and exception cleanup have reproducible evidence.
   - [ ] Nested projection, Variant extraction, schema/time boundaries, storage 
and incremental contexts have explicit supported/fallback matrices.
   - [ ] BE/FE builds and applicable unit/regression tests run on the final 
candidate; limitations are stated accurately.
   
   The Native-priority change is an independently reviewable first step. It 
must not be presented as completion of this integration issue, and the 
resource-sensitive gates must stay until their dependencies are resolved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to