HappenLee opened a new issue, #68239:
URL: https://github.com/apache/doris/issues/68239
### Search before asking
Searched existing issues for commit TSO and ordered/link compaction. Related
reader and pruning reports are #67994 and #67995. This report concerns loss of
the per-segment TSO interpretation during compaction, which is a separate
problem.
### Version
Apache Doris master, source baseline
`84ebcf71bb52735c22034d9185c51e373caa9acd`.
This is a source-derived correctness report. The end-to-end reproduction
described below has not been executed for this report.
### What's Wrong?
Ordered/link compaction can turn several single-version rowsets of a local
DUPLICATE KEY table with ROW binlog enabled into a multi-version rowset without
materializing their logical commit TSO values.
For a published single-version rowset, `__DORIS_COMMIT_TSO_COL__` contains a
physical placeholder of `0`. Its logical value comes from that rowset's commit
TSO metadata. Ordered compaction links the existing segment files, combines the
version interval, and records an overall commit TSO range. It does not preserve
which source TSO belongs to each linked segment.
The reader substitutes a constant TSO only for a single-version rowset. Once
the output spans multiple versions, it reads the physical column instead. The
linked files still contain `0`.
For example, using illustrative metadata values:
```text
Before compaction:
Rowset A: version=[7,7], commit_tso=[1000,1000], physical TSO=0
Rowset B: version=[8,8], commit_tso=[2000,2000], physical TSO=0
Logical row values: 1000 for A, 2000 for B
After linking the files:
Output: version=[7,8], commit_tso=[1000,2000]
Physical row values are still 0
The single-version substitution no longer applies
```
Consequences inferred from these code paths:
- Projecting the hidden column can return `0` instead of the original commit
TSO.
- A `commit_tso > 0` predicate can incorrectly reject matching rows.
- A `commit_tso <= 1500` predicate can incorrectly accept rows from B,
including when placeholder ZoneMaps make the predicate appear always true.
- Historical reads relying on these boundaries can return an incorrect row
set.
Disabling ZoneMap pruning alone cannot restore the missing logical values.
The scope is the local ordinary DUPLICATE KEY base-table ordered-compaction
path. A base table with ROW binlog enabled is not the dedicated row-binlog
tablet. The examined admission path rejects Cloud mode and ordinary
merge-on-write tablets; this report does not claim those paths are affected by
this specific mechanism.
### What You Expected?
Compaction must preserve the logical commit TSO of every row. In the
example, `commit_tso > 0` must retain both input batches, while `commit_tso <=
1500` must retain only A's rows. Projection, filtering, and historical reads
must agree before and after compaction.
### How to Reproduce?
The following is a proposed BE fixture or controlled-cluster validation
procedure, not a claimed successful SQL reproduction:
1. Create a local DUPLICATE KEY table with ROW binlog enabled so its schema
includes `__DORIS_COMMIT_TSO_COL__`.
2. Produce two published single-version rowsets with distinct positive
commit TSOs and record the expected TSO for each row from the source rowset
metadata.
3. Ensure the normal ordered-compaction admission conditions hold: matching
schema versions, strictly ordered non-overlapping key ranges, sufficiently
large segments, and no other conditions forcing a rewrite. Enable
`enable_ordered_data_compaction`.
4. Trigger cumulative or base compaction. Verify through instrumentation or
logs that `handle_ordered_data_compaction()` actually links the target input
segments. Merely completing compaction is insufficient to exercise the bug.
5. Open fresh readers on the output rowset and compare projected commit TSOs
and the two boundary predicates above with the saved oracle. Compare the actual
row identities, not only row counts.
6. Repeat with ordinary rewrite compaction as a control, then validate
equivalent historical-read boundaries.
### Anything Else?
Source references at the examined baseline:
- [Ordered-compaction
admission](https://github.com/apache/doris/blob/84ebcf71bb52735c22034d9185c51e373caa9acd/be/src/storage/compaction/compaction.cpp#L571).
- [Linking segment files and building the output
metadata](https://github.com/apache/doris/blob/84ebcf71bb52735c22034d9185c51e373caa9acd/be/src/storage/compaction/compaction.cpp#L461).
- [Single-version commit TSO
substitution](https://github.com/apache/doris/blob/84ebcf71bb52735c22034d9185c51e373caa9acd/be/src/storage/segment/segment.cpp#L947).
A minimal fix direction is to exclude ordinary base-table rowsets that rely
on commit TSO placeholders from ordered/link compaction and use the normal
rewrite path. Retaining the linking optimization would require preserving the
source-segment-to-TSO mapping and teaching readers to use it, or otherwise
materializing the real values. An overall minimum/maximum TSO range is
insufficient.
Related PR: #68097 resolves logical commit TSO readers when the source
metadata is available. Its single-version rule cannot reconstruct the lost
mapping in this multi-version output, and it does not change the
ordered-compaction path. This failure mode predates that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
