eldenmoon opened a new pull request, #66910:
URL: https://github.com/apache/doris/pull/66910
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Iceberg unshredded Variant files store each root as `metadata + value`. Path
extraction previously materialized the complete root `ColumnVariantV2` before
evaluating `element_at`, even when a query needed only one nested path. This
made JSONBench path-heavy queries spend most of their scan time reconstructing
values that were immediately discarded.
This change keeps the physical unshredded representation and uses the
existing Variant seek primitives to:
- seek a requested object/array path directly in the encoded value;
- reuse the located prefix for chained `element_at` calls;
- return a typed STRING leaf when every present value in the batch is a
string;
- cache repeated path results across filter/range/index selections;
- preserve SQL NULL, missing path, and Variant JSON null as distinct states;
- expose direct-seek, prefix-reuse, result-cache, subtree, and copied-byte
counters.
The fast path is enabled only for the exact two-child unshredded `metadata +
value` layout. This PR intentionally contains no shredded Variant optimization
and does not bypass typed/residual overlay semantics.
#### Performance validation
The comparison used the same Iceberg metadata and 1,000 Parquet files
(`999,999,258` rows, `124,084,905,328` bytes) for Doris and StarRocks. Both
engines were pinned to logical CPUs `0-29`; every sample waited until both
whole-host and selected-CPU idle were greater than 70%. Builds were Release,
session semantics and result ordering were matched, and every query used one
warmup plus five timed repetitions.
The "optimization disabled" column is a reverse A/B of the same Doris source
with only the two direct-seek entry conditions disabled, followed by a rebuild.
The optimized source was then restored, rebuilt, and smoke-checked again.
| Query | Doris disabled median | Doris optimized median / p95 | StarRocks
median / p95 | Optimized vs disabled | Doris vs StarRocks |
| --- | ---: | ---: | ---: | ---: | ---: |
| Q1 | 123.121 s | 41.950 / 42.038 s | 48.755 / 50.131 s | 2.93x | 13.96%
faster |
| Q2 | 312.404 s | 77.874 / 77.995 s | 86.431 / 89.903 s | 4.01x | 9.90%
faster |
| Q3 | 348.944 s | 82.999 / 83.034 s | 157.360 / 170.855 s | 4.20x | 47.26%
faster |
| Q4 | 218.528 s | 66.182 / 66.493 s | 76.105 / 76.348 s | 3.30x | 13.04%
faster |
| Q5 | 221.837 s | 66.458 / 66.729 s | 75.747 / 77.325 s | 3.34x | 12.26%
faster |
All Q1-Q5 result SHA-256 hashes match between optimized Doris, disabled
Doris, and StarRocks.
For the representative Q2 profile, disabling the optimization reports
`1.947B` reconstructed rows, zero direct-seek rows, and about 23 minutes of
aggregate reconstruction time. The optimized profile reports zero reconstructed
rows, `4.894B` direct-seek rows, `1.947B` prefix-reuse rows, and `3.788B`
result-cache-hit rows. These counters are summed across scanner instances and
repeated path evaluations, so they can exceed the table row count.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `./run-be-ut.sh --run --filter=VariantColumnReaderTest.*` (42
passed)
- [x] Manual test
- `PATH=/mnt/disk6/common/ldb_toolchain_toucan/bin:$PATH
build-support/clang-format.sh`
- `PATH=/mnt/disk6/common/ldb_toolchain_toucan/bin:$PATH
build-support/check-format.sh`
- `./build.sh --be --fe` with `BUILD_TYPE=RELEASE`
- final optimized `./build.sh --be` with `BUILD_TYPE=RELEASE`
- 1B-row Iceberg JSONBench Q1-Q5, optimized/disabled reverse A/B,
one warmup plus five timed repetitions
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Unshredded Parquet Variant path extraction avoids
complete-root reconstruction; query results and null semantics are unchanged.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
