eldenmoon opened a new pull request, #68444:
URL: https://github.com/apache/doris/pull/68444
### What problem does this PR solve?
Issue Number: None
Related PR: #56815
Problem Summary:
`information_schema.column_data_sizes` reports 0 for COMPRESSED_DATA_BYTES,
UNCOMPRESSED_DATA_BYTES and RAW_DATA_BYTES of every VARIANT column, so a
per-column size breakdown leaves the variant out and overstates the share of
the other columns. Reproduce on master with 200,000 rows whose `js` STRING
column and `v` VARIANT column hold the same JSON:
```sql
CREATE TABLE t_cds (id BIGINT, js STRING, v VARIANT) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t_cds SELECT number,
concat('{"a":', number, ',"b":"', repeat('y', 50), '"}'),
parse_to_variant(concat('{"a":', number, ',"b":"', repeat('y', 50),
'"}'))
FROM numbers("number" = "200000");
SELECT c.COLUMN_NAME, SUM(c.COMPRESSED_DATA_BYTES)
FROM information_schema.column_data_sizes c
JOIN information_schema.metadata_name_ids m ON c.TABLE_ID = m.TABLE_ID
WHERE m.TABLE_NAME = 't_cds' GROUP BY 1;
-- id 8214, js 823161, v 0 (still 0 after a full compaction)
```
The table reads the data page sizes that the segment writer stores in each
column meta of the segment footer. Two gaps drop the variant:
1. `VariantColumnWriter`, `VariantSubcolumnWriter` and
`VariantDocCompactWriter` return 0 (TODO) from `get_raw_data_bytes()`,
`get_total_uncompressed_data_pages_bytes()` and
`get_total_compressed_data_pages_bytes()`, although the variant data is
written by their inner writers: the root, the materialized subcolumns and
the sparse or doc value columns.
2. Compaction writes variant subcolumns, sparse columns and doc value columns
as columns of their own. Their metas have no unique id, so the scanner
drops
them; with external column meta (`storage_format = "V3"`) the sparse and
doc value metas are also embedded in the root variant meta, which the
scanner never looks into.
Fix:
- The variant writers sum the three counters over their inner writers, the
same way the struct, array and map writers already do.
`ColumnWriter::DataBytesGetter` names the getter that the variant writer
layers pass down.
- The scanner counts a column meta that has data bytes but no unique id to
its
parent variant column (from its column path info), adds the sparse and doc
value metas embedded in a variant root meta, and takes COLUMN_TYPE from the
schema column (a subcolumn meta carries the subcolumn's type). Nothing is
counted twice: the inner metas of a variant writer carry no data bytes.
After the fix, the variant reports the size of all its data pages before and
after compaction, for inline and external column meta, in ordinary and doc
mode. Segments written before the fix keep their old numbers until compaction
rewrites them.
The segment footer now stores real data bytes on variant column metas instead
of 0, so the 29 checked-in golden Segment files of the 14 variant cases in
`segment_flusher_format_test` are regenerated (each grows by 2 to 4 bytes;
only those three counters of the variant metas differ). The other 59 golden
cases are byte-identical.
The footer raw_data_bytes of a variant root is also the first-batch hint of
the
segment iterator's adaptive batch size predictor when a query reads the whole
variant column. It was 0 for variants; it now holds the variant's data for
segments written by a load and the root's data for compaction output. It only
affects the first batch, which is capped at the probe rows either way.
Vertical
compaction's footer-based batch size estimate still skips variant columns;
its
stale comment is updated.
### Release note
Fix `information_schema.column_data_sizes` reporting 0 bytes for VARIANT
columns. Data written before the fix reports the new sizes once compaction
rewrites it.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
BE UT: the variant column writers report exactly the data page bytes
they append to the segment file (ordinary and doc mode variant writers,
extracted subcolumn writer, doc compact writer); the golden segment tests,
variant, compaction and segment writer tests pass.
Regression test `test_query_sys_column_data_sizes_variant`: ordinary and
doc mode variants with V2 and V3 column meta, before and after full compaction.
The data pages of all columns must lie within the rowsets' DATA_DISK_SIZE and
cover more than half of it; they cover 91%-96%. Without counting the metas
embedded in the variant root, the V3 tables drop to 35% and 11% after
compaction; without resolving the parent unique id, the V2 tables drop to 3%
and 2%.
- Behavior changed:
- [ ] No.
- [x] Yes. `information_schema.column_data_sizes` reports the data size
of VARIANT columns instead of 0, and variant column metas in the segment footer
store their data page bytes instead of 0.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
