suxiaogang223 opened a new issue, #70797: URL: https://github.com/apache/doris/issues/70797
### Background Doris can read safe raw Paimon files with its Native ORC/Parquet readers and use a Paimon SDK for logical reads. The optional Rust reader currently precedes Native in compatible ordinary scans. Its compatibility gates also mix actual SDK restrictions, incomplete C ABI/host integration, and assumptions that predate paimon-rust 0.5.0. ### Expected behavior Preserve accurate metadata COUNT, then choose Native > Rust > JNI. `force_jni_scanner` still overrides data readers. Enabling Rust should prefer it where a Paimon logical reader is needed, while preserving Native correctness checks, footer aggregates, deletion vectors, diagnostic split suppression, and mixed-backend capability checks. ### Scope Align ordinary queries, including the incremental query forms already exposed by Doris, against paimon-rust v0.5.0 code `bb6389a8b430e897e038ff7395b6a7763d785a7a` and Java Paimon 2.1.0. Do not infer 0.5.0 support from Rust main/0.6. Writes and dedicated full-text/vector/hybrid optimizer pushdown are separate work. ### Proposed implementation and dependencies 1. Evaluate Native eligibility completely before considering Rust, including historical ORC and cross-family timestamp checks. Update the old priority assertions. 2. Establish a query-shared resource and cancellation contract before lifting multi-file primary-key or partial-update/aggregation gates. Include retained merge inputs, same-key versions, aggregate state, pending Arrow output, concurrent splits, exhaustion, and cleanup. Small batches and `with_resources` alone are insufficient. 3. Rebuild the function × type × option matrix against the fixed source. Integer sum/product wrap by default and collect/map/nested/sketch functions exist; retaining an outdated SDK explanation is incorrect. Keep actual unsupported Java SPI or semantic combinations on JNI. 4. Adapt recursive projections and Variant carriers across C/Arrow/Doris. The core read-type API is not exposed by the current root-name C projection API; Rust's Variant child order is value/metadata while Doris's native Arrow carrier expects metadata/value. Do not remove gates without a correct mapping and mixed-BE negotiation. 5. Verify schema evolution, timestamp range/timezones, DV/MOR, external paths, credentials/FileIO/cache, and existing incremental query semantics. Retain concrete fallback reasons for unintegrated contracts. SDK dependencies include apache/paimon-rust#1386 (pending-output batch retention), apache/paimon-rust#1468 (reader budget coverage and C host-pool integration), and apache/paimon-rust#1382 (audit/incremental C execution). A patched SDK needs a tested 0.5.0-compatible backport and a matching thirdparty prebuilt; changing to a newer SDK version is not equivalent to integrating the pinned version. ### Evidence so far On Doris master `5bc6ed62f9b01cd2cecaf9a3420d5ee015638ec1`, the Native-first candidate passes 148 FE tests: 128 scan-provider, 12 Rust-selector, 8 capability tests. The new raw-file priority assertion fails against the original provider and passes against the candidate. Independent C ABI probes using the 2026-11-19 Doris prebuilt matched Java 2.1.0 ordered results for four merge-engine controls in Parquet and ORC. These are SDK probes, not proof that Doris executed its Rust reader. Resource probes reproduced a gap: ORC reads succeed with a zero-byte resource context and report zero peak reservations. For two overlapping files with one early survivor and 32768 subsequent deletes, Parquet's reported peak reservation stays 44020 bytes under a 2 MiB budget, while process peak RSS changes from 27236 to 95996 KiB when read.batch-size changes from 1 to 32. RSS includes runtime/decoder overhead; the exact retained-state accounting still needs deterministic tests. Batch size 1 is a control, not a fix. ### Acceptance criteria - [ ] Native priority with COUNT/footer optimizations, complete Native time checks, force-JNI, ignore_split_type and mixed-BE coverage. - [ ] Real non-raw-convertible tasks execute Doris Rust, confirmed by completed Profiles and ordered JNI comparisons; raw append scans remain Native. - [ ] Four merge engines, updates/deletes, multiple files/levels, DV, sequence/aggregation options and invalid combinations have result/error coverage. - [ ] Shared memory budget, oversized same-key state, sparse output, concurrency, cancellation and exception cleanup have reproducible evidence. - [ ] Nested projection, Variant extraction, schema/time boundaries, storage and incremental contexts have explicit supported/fallback matrices. - [ ] BE/FE builds and applicable unit/regression tests run on the final candidate; limitations are stated accurately. The Native-priority change is an independently reviewable first step. It must not be presented as completion of this integration issue, and the resource-sensitive gates must stay until their dependencies are resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
