eldenmoon commented on PR #70408:
URL: https://github.com/apache/doris/pull/70408#issuecomment-6998961193
[P1] Non-atomic RESTORE can cause compaction to drop VARIANT fields
Restoring a V1 count=0 backup into an existing V2 count=0 table can leave
both layouts at the same schema version. If compaction picks the V1 schema, it
can mark a sparse-only path for extraction without creating a subcolumn for it.
The sparse merge then drops the value. For example, after restoring {"a":1} and
inserting {"b":2}, compaction can lose b=2 while preserving the row count.
The current guard only covers V2 targets, so this can happen on a fully
upgraded cluster. Please handle the V1-target case before merge and add a
RESTORE → INSERT → compaction regression test that checks the field value.
[Compaction
code](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/be/src/exec/common/variant_util.cpp#L1291-L1335)
[P1] Persisted SQL can fail to reparse after upgrade
Existing Routine Load jobs, views, and materialized views can contain
VARIANT expressions with variant_enable_typed_paths_to_sparse=true,
variant_doc_materialization_min_rows, or variant_doc_hash_shard_count. The new
parser rejects these forms. Replaying a Routine Load job can set it to
CANCELLED, and view queries or MV refreshes can fail when Doris reparses their
definitions.
Please preserve compatibility for persisted SQL before merge and add a
replay test using an old definition. [Routine Load
replay](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L2109-L2163)
[P2] Compaction applies the extraction threshold to input rows
For ordinary VARIANT columns, compaction compares
variant_subcolumn_extraction_min_rows with the sum of input rowset rows. With a
threshold of 100, it can still extract subcolumns when 120 input rows
deduplicate to 60 output rows. Splitting the output across segments can produce
the same mismatch. This changes when extraction happens and can increase write
and storage costs. The existing review comment on this behavior still applies.
[Threshold
calculation](https://github.com/apache/doris/blob/7583d8b979103817dca8deb070964adbd520992b/be/src/exec/common/variant_util.cpp#L1229-L1238)
[Other] Mixed-version writes with a lowered be_exec_version
The default be_exec_version=19 prevents 4.0 and 4.1 BEs from accepting
normal load blocks. However, the FE still creates V2 columns when that setting
is lowered. If a load reaches an older BE with a compatible execution version,
the BE can ignore variant_serialize_type and write count=0 using V1 semantics.
Is this configuration supported? If so, please either prevent older BEs from
writing V2 count=0 columns or add a mixed-version load test that verifies
compaction preserves sparse-only fields when it selects the V1 schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]