github-actions[bot] commented on code in PR #70408:
URL: https://github.com/apache/doris/pull/70408#discussion_r4596846325
##########
be/src/exec/common/variant_util.cpp:
##########
@@ -857,8 +900,14 @@ Status
VariantCompactionUtil::aggregate_variant_extended_info(
// get the subpaths and sparse paths for the variant column
void VariantCompactionUtil::get_subpaths(int32_t max_subcolumns_count,
+ VariantSerializeTypePB serialize_type,
const PathToNoneNullValues& stats,
VariantCompactionPaths&
paths_set_info) {
+ // Count 0 means no limit under V1, and no dynamic subcolumn under V2
(every path stays sparse).
+ if (max_subcolumns_count == 0 &&
+ serialize_type == VariantSerializeTypePB::VARIANT_SERIALIZE_V2) {
+ return;
Review Comment:
[P1] Fold historical extracted paths into the V2 sparse output. A new FE can
create a V2 count-zero column on an old BE, which ignores the new optional
field and stores it as V1 with every dynamic path extracted. Insert {"a":1},
upgrade that BE, then perform a light ADD COLUMN and load a different path such
as {"b":2}: the higher-version V2 rowset becomes the compaction target. Old
segments still open with their own V1 schema, so
check_variant_serialize_type_layout does not reject them. Returning here leaves
sparse_path_set empty: _create_sparse_merge_reader skips a, while the V2 output
schema creates no dynamic subcolumns and ROOT_ONLY does not reconstruct object
children. Since the statistics check only visits output paths, it also misses
the disappearance of a. Compaction can therefore silently discard that
committed value. Populate sparse_path_set so historical extracted paths are
folded into the sparse output, and cover V1 input rowsets compacted into a V2
count-zero target.
##########
be/src/exec/common/variant_util.cpp:
##########
@@ -1179,6 +1228,13 @@ Status
VariantCompactionUtil::get_extended_compaction_schema(
}
}
+ // The rows this compaction writes, compared with
variant_subcolumn_extraction_min_rows below.
+ // A load decides per segment; here the whole output stands for its
segments.
+ int64_t input_rows = 0;
+ for (const auto& rs : rowsets) {
+ input_rows += rs->num_rows();
Review Comment:
[P2] Base deferred extraction on output segment rows. This sum counts rows
before compaction merges or filters them, and it also combines rows that may go
into separate output files. For example, with
variant_subcolumn_extraction_min_rows=5, two merge-on-read UNIQUE KEY rowsets
containing the same three keys stay sparse when loaded, but this code selects
subcolumns from six input rows even though compaction writes only three rows.
The ordinary writer then takes its root-only path and writes the selected
subcolumns without invoking select_variant_paths, so the actual file size never
corrects the decision. Use output segment row counts (and fold paths back into
sparse below the threshold), with coverage for duplicate elimination and output
splitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]