morningman opened a new pull request, #70767:
URL: https://github.com/apache/doris/pull/70767

   ### Background
   
   Parquet can store Variant fields as typed columns alongside encoded residual 
values. Doris can project individual fields, using the root metadata dictionary 
to decode residual keys. Row-group statistics can prove that residual columns 
are entirely NULL.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #66941 (same change on 4.1/4.2), #68597 (STRING profile 
stabilization), #66413 (introduced the reader), #69016 (deep leaf projection), 
#70664 (scan splitting)
   PR Type: Bugfix
   
   Problem Summary:
   
   This ports #66941 and #68597 to master. Complete STRING leaves and mixed 
residual batches unnecessarily rebuild Variant roots; a NULL typed root can 
also hide a requested value stored in its residual. Resolve rows from their 
typed leaf or the first residual that owns the path, and prune unused physical 
columns. Most added lines are tests.
   
   Original authors: @hubgeter and @Gabriel39.
   
   ### Details
   
   - **Change map:** Production +682/-156 lines in 9 files; tests +1400/-88 in 
8 files. The production parts are leaf/residual decoding and validation 
caching, mapper/row-group projection and byte accounting, and profile/composite 
validation. No generated outputs or docs.
   - **Frequency:** Deterministic for the affected complete STRING, 
mixed-residual and NULL typed-root shapes.
   - **Root cause:** The path walk ignores the root residual. Complete-state 
scalar guards and batch-wide residual rejection force fallback; the mapper 
excludes scalar identities the reader can normalize.
   - **Why existing tests missed it:** Existing BE tests exercised 
reconstructed values and conservative fallback. Iceberg/Paimon regressions 
checked values and BIGINT projection, without complete STRING direct access, 
row-level residual ownership, shared validation or pruned physical-column 
assertions.
   - **Fix:** Carry metadata and terminal residuals with scalar projections, 
then finalize them per row group. Preserve typed UTF-8 leaves; normalize other 
scalar identities through the Parquet schema. Cache dictionary field IDs and 
residual validation, and validate every composite segment. Preserve master's 
UUID/Decimal128 rules and shared profile ownership.
   - **Other paths and branches:** Iceberg and native Paimon share this reader; 
both suites are covered. Complex/repeated projections retain complete fallback. 
Branches 4.1/4.2 already have the change. Local 3.1/4.0 refs have no V2 reader 
here and were not refreshed. Statistics filtering retains its conservative 
ancestor-residual rule.
   - **Behavior change:** Add `VariantDirectLeafResidualMergedRows`, 
`VariantResidualSeekRows` and `VariantResidualProjectionRowGroupColumns` to 
profiles. Configs and defaults are unchanged.
   
   ### Testing
   
   - Local macOS arm64, LLVM22, ASAN: 494 cases across the six BE files; 491 
passed, two optional microbenchmarks skipped, one preexisting comparison test 
failed. All 28 newly ported cases passed without skips. A temporary focused 
test build included their support sources and was restored afterward.
   - Restoring master's original lookup/composite methods makes four new 
controls fail at the intended assertions. The existing 
`TypedComparisonBoundariesMatchEncodedWithoutMaterializing` repeats twelve 
sign-versus-magnitude failures with those original methods; it is left 
unchanged for a separate test-only fix.
   - Original Iceberg and Paimon Variant suites both passed locally, each with 
zero skipped scripts, using Spark 4, MinIO and Iceberg REST with SQLite. The 
4096-row STRING query reports one leaf projection, zero full/residual 
projections and zero reconstructed rows. No timing speedup claim.
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: the existing macOS comparison 
assertion fails independently; full Linux BE/unit validation was not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: adaptation, tests, verification and this description; the main 
implementation and tests originate from the related PRs.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to