eldenmoon opened a new pull request, #67016:
URL: https://github.com/apache/doris/pull/67016
### What problem does this PR solve?
Issue Number: None
Related PR: #66910
Problem Summary:
Iceberg unshredded Variant files store each root as `metadata + value`. Path
extraction previously materialized the complete root `ColumnVariantV2` before
evaluating `element_at`, even when a query needed only a small scalar path.
This change keeps the complete physical unshredded representation and:
- seeks requested object and array paths directly in the encoded value;
- reuses located prefixes for chained `element_at` calls;
- deduplicates metadata dictionaries per physical state and resolves object
field IDs once per dictionary and path;
- returns typed STRING and canonical integer leaves while preserving
explicitly widened integers on the encoded path;
- preserves SQL NULL, missing path, and Variant JSON null as distinct states;
- falls back to a lazily materialized encoded subtree for mixed or
unsupported scalar paths;
- exposes direct-seek, prefix-reuse, subtree, typed-leaf, fallback, and
copied-byte counters.
The final design does not retain a result cache. It caches only the metadata
index and the located prefix needed by chained extraction, reducing state and
invalidation complexity.
The fast path is enabled only for the exact complete two-child unshredded
`metadata + value` layout. It does not directly seek shredded residual values
or bypass typed/residual overlay semantics. For valid input, SQL results and
null semantics are unchanged. The direct path validates the traversed object
layout and the metadata required by optimized integer output; it does not
proactively scan unrelated, unvisited payload regions before returning a
selected leaf.
#### Performance validation
The following measurements were collected on the branch-4.1 implementation
from which this master port was prepared. They are retained as historical
performance evidence; this master commit has not been re-benchmarked
independently.
##### 1B-row matched-layout benchmark
The direct-seek baseline `8bd41ac179f5` used the same Iceberg metadata and
1,000 Parquet files (`999,999,258` rows, `124,084,905,328` bytes) for Doris and
S system. Both systems were pinned to logical CPUs `0-29`; every sample waited
until both whole-host and selected-CPU idle were greater than 70%. Builds were
Release, session semantics and result ordering were matched, and every query
used one warmup plus five timed repetitions.
| Query | Doris disabled median | Doris optimized median / p95 | S system
median / p95 | Optimized vs disabled | Doris vs S system |
| --- | ---: | ---: | ---: | ---: | ---: |
| Q1 | 123.121 s | 41.950 / 42.038 s | 48.755 / 50.131 s | 2.93x | 13.96%
faster |
| Q2 | 312.404 s | 77.874 / 77.995 s | 86.431 / 89.903 s | 4.01x | 9.90%
faster |
| Q3 | 348.944 s | 82.999 / 83.034 s | 157.360 / 170.855 s | 4.20x | 47.26%
faster |
| Q4 | 218.528 s | 66.182 / 66.493 s | 76.105 / 76.348 s | 3.30x | 13.04%
faster |
| Q5 | 221.837 s | 66.458 / 66.729 s | 75.747 / 77.325 s | 3.34x | 12.26%
faster |
All Q1-Q5 result SHA-256 hashes matched between optimized Doris, disabled
Doris, and S system.
##### Latest hot-cache benchmark
This run used a different workload and cache protocol from the 1B-row
benchmark, so its absolute times should not be compared directly with the first
table.
| Query | Doris | S system | Doris relative performance |
| --- | ---: | ---: | ---: |
| Q1 | 0.330 s | 0.277 s | 19.1% slower |
| Q2 | 6.748 s | 9.068 s | 25.6% faster |
| Q3 | 5.776 s | 7.868 s | 26.6% faster |
| Q4 | 6.992 s | 10.034 s | 30.3% faster |
| Q5 | 0.057 s | 0.077 s | 26.0% faster |
| Q6 | 0.070 s | 0.040 s | 75.0% slower (30 ms absolute) |
| Q7 | 4.365 s | 5.870 s | 25.6% faster |
| Q8 | 60.807 s | 64.252 s | 5.4% faster |
| Q9 | 4.771 s | 6.387 s | 25.3% faster |
| Q10 | 34.357 s | 70.080 s | 51.0% faster |
The hot-cache results include the metadata-index and integer-leaf paths. A
new matched 1B-row reverse A/B would still be required to isolate their
incremental gain over the earlier direct-seek baseline.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- Attempted `./run-be-ut.sh --run
--filter='VariantColumnReaderTest.*:VariantValueTest.*'` (ASAN). The changed
translation units compiled, but the final test-binary link was blocked before
test execution by duplicate `getrandom` symbols from the local
glibc-compatibility libraries.
- [x] Manual test
- `build-support/clang-format.sh`
- `build-support/check-format.sh`
- `git diff --check origin/master`
- Historical branch-4.1 Release 1B Iceberg JSONBench Q1-Q5,
optimized/disabled reverse A/B, one warmup plus five timed repetitions
- Historical branch-4.1 hot-cache Q1-Q10 benchmark: Doris vs S system
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Complete unshredded Parquet Variant path extraction avoids
redundant CPU work for valid input; SQL results and null semantics are
unchanged.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
