morningman opened a new pull request, #70767: URL: https://github.com/apache/doris/pull/70767
### Background Parquet can store Variant fields as typed columns alongside encoded residual values. Doris can project individual fields, using the root metadata dictionary to decode residual keys. Row-group statistics can prove that residual columns are entirely NULL. ### What problem does this PR solve? Issue Number: None Related PR: #66941 (same change on 4.1/4.2), #68597 (STRING profile stabilization), #66413 (introduced the reader), #69016 (deep leaf projection), #70664 (scan splitting) PR Type: Bugfix Problem Summary: This ports #66941 and #68597 to master. Complete STRING leaves and mixed residual batches unnecessarily rebuild Variant roots; a NULL typed root can also hide a requested value stored in its residual. Resolve rows from their typed leaf or the first residual that owns the path, and prune unused physical columns. Most added lines are tests. Original authors: @hubgeter and @Gabriel39. ### Details - **Change map:** Production +682/-156 lines in 9 files; tests +1400/-88 in 8 files. The production parts are leaf/residual decoding and validation caching, mapper/row-group projection and byte accounting, and profile/composite validation. No generated outputs or docs. - **Frequency:** Deterministic for the affected complete STRING, mixed-residual and NULL typed-root shapes. - **Root cause:** The path walk ignores the root residual. Complete-state scalar guards and batch-wide residual rejection force fallback; the mapper excludes scalar identities the reader can normalize. - **Why existing tests missed it:** Existing BE tests exercised reconstructed values and conservative fallback. Iceberg/Paimon regressions checked values and BIGINT projection, without complete STRING direct access, row-level residual ownership, shared validation or pruned physical-column assertions. - **Fix:** Carry metadata and terminal residuals with scalar projections, then finalize them per row group. Preserve typed UTF-8 leaves; normalize other scalar identities through the Parquet schema. Cache dictionary field IDs and residual validation, and validate every composite segment. Preserve master's UUID/Decimal128 rules and shared profile ownership. - **Other paths and branches:** Iceberg and native Paimon share this reader; both suites are covered. Complex/repeated projections retain complete fallback. Branches 4.1/4.2 already have the change. Local 3.1/4.0 refs have no V2 reader here and were not refreshed. Statistics filtering retains its conservative ancestor-residual rule. - **Behavior change:** Add `VariantDirectLeafResidualMergedRows`, `VariantResidualSeekRows` and `VariantResidualProjectionRowGroupColumns` to profiles. Configs and defaults are unchanged. ### Testing - Local macOS arm64, LLVM22, ASAN: 494 cases across the six BE files; 491 passed, two optional microbenchmarks skipped, one preexisting comparison test failed. All 28 newly ported cases passed without skips. A temporary focused test build included their support sources and was restored afterward. - Restoring master's original lookup/composite methods makes four new controls fail at the intended assertions. The existing `TypedComparisonBoundariesMatchEncodedWithoutMaterializing` repeats twelve sign-versus-magnitude failures with those original methods; it is left unchanged for a separate test-only fix. - Original Iceberg and Paimon Variant suites both passed locally, each with zero skipped scripts, using Spark 4, MinIO and Iceberg REST with SQLite. The 4096-row STRING query reports one leaf projection, zero full/residual projections and zero reconstructed rows. No timing speedup claim. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the existing macOS comparison assertion fails independently; full Linux BE/unit validation was not run. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: adaptation, tests, verification and this description; the main implementation and tests originate from the related PRs. - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
