HappenLee opened a new pull request, #68050:
URL: https://github.com/apache/doris/pull/68050
### What problem does this PR solve?
Issue Number: None
Related PR: #68012 (merged into `branch-incremental-computation`); this PR
ports the same change to `master`.
Problem Summary:
Bounded ROW binlog queries initialize rowset readers and open segment
footers before their TSO predicates reject historical data. A narrow time
window can therefore pay initialization costs proportional to the retained
history.
Prune the captured read source before cloning readers or creating scanners.
For query `[100, 200)`, a rowset with inclusive commit TSO range `[10, 99]` can
be skipped using its metadata alone.
- Preserve rowsets with unknown TSO endpoints and those that overlap the
query window, including compacted rowsets. Existing segment/row predicates
handle the remaining data.
- Preserve the captured visible-version snapshot and separately captured
delete predicates.
- Skip a tablet when only empty rowsets remain, including bootstrap rowsets
with no TSO. This also prevents a fully pruned source from being recaptured
during scanner initialization.
- Add `RowsetTsoPruneTime`, `RowsetsPrunedByTso`, `SegmentsPrunedByTso`, and
`TabletsPrunedByTso` to the query profile.
The filtering uses `std::erase_if` on the existing rowset vector and
`std::all_of` for the empty-tablet check. It adds no storage-format or protocol
changes.
### Release note
Reduce reader initialization and segment footer IO for bounded ROW binlog
queries.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (steps and results below)
- [ ] No need to test or manual test.
Validation:
Current master port (base `060c7dbe81758fb0d673567fb361234d06b9a1b1`):
- Applied commit `93a3ceced9c4518d527c2b2d58ee328bb5960d03` without
conflicts. Stable patch IDs match the source PR exactly.
- Repository `build-support/check-format.sh` passed with clang-format 16.0.6.
- `build-support/check-build-hygiene.sh` passed.
- Source whitespace checks passed. The original, runner-generated regression
`.out` is preserved unchanged, including its trailing blank line.
- Builds, unit tests, regression tests, and clang-tidy have not been rerun
on this master base.
Historical validation reported in #68012 for the original master-based
implementation (`cb73d23c873`, based on `96d0ac68e84`):
- BE (ASAN) and FE built successfully with `build.sh --be --fe -j 48`.
- All 8 tests in `OlapScanOperatorTsoPruningTest` and
`OlapScanOperatorBinlogPushDownTest` passed under ASAN. Coverage includes
half-open boundaries, single/no bounds, unknown metadata, overlapping
compaction output, delete predicates, and empty-source EOS.
- `test_binlog_rowset_tso_pruning` and `test_binlog_changes_syntax` passed.
The new output was generated by the regression runner and then checked in a
second run. Before-images, deletes, empty windows, and compaction results were
verified.
- clang-format 16, header hygiene, and clang-tidy passed.
Manual profile check: create a one-bucket UNIQUE KEY table with ROW binlog,
historical values, and automatic compaction disabled; insert two rows in two
separate transactions. Enable profiling and query
`@incr('startTimestamp'='2099-01-01 00:00:00', 'incrementType'='MIN_DELTA')`.
The empty result reported:
```text
RowsetsPrunedByTso: 2
SegmentsPrunedByTso: 2
TabletsPrunedByTso: 1
NumScanners: 0
RowsRead: 0
```
This verifies that scanner initialization is skipped. No production latency
improvement is claimed; rowset metadata synchronization and version capture
still occur.
- Behavior changed:
- [x] No. Query results are unchanged; profile counters are added.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
