Yukang-Lian opened a new pull request, #65470:
URL: https://github.com/apache/doris/pull/65470
### What problem does this PR solve?
Issue Number: None
Related PR: #59268, #64954
Problem Summary:
#### Problem
Under memory pressure, cloud cumulative compaction reduces the maximum
compaction score. A production-shaped MOW tablet can then contain a
zero-segment, non-overlapping compaction output at the cumulative point,
followed by large overlapping singleton rowsets whose individual scores exceed
that reduced maximum.
The max-score back trim introduced by #59268 can remove every successor from
the original selected set while retaining the non-overlapping head. The caller
rejects that single rowset with `CUMULATIVE_NO_SUITABLE_VERSION`. Because no
merge runs, the previous memory-limit status remains in effect, the same
reduced maximum is reused, and the tablet can repeatedly hit the same
no-suitable-version result while new versions accumulate.
The cloud scheduling loop also suppressed this status outside debug logging,
which made the retry loop difficult to observe.
#### Root cause
The deferred max-score trim deliberately keeps at least one rowset, but it
did not verify that the remaining input was mergeable. When the head was
non-overlapping and its direct successor alone exceeded the remaining score
budget, trimming could strand the head as an invalid single-rowset input.
#### Fix
- In both cloud and local size-based cumulative compaction policies,
remember the most recently removed rowset while trimming from the back.
- If trimming leaves exactly one non-overlapping rowset, restore that direct
successor and accept a one-off max-score overshoot. Restore both score and size
accounting, and do not trim again.
- Keep the cloud empty-rowset `skip_trim` behavior unchanged.
- Leave cumulative-point and meta-service job handling unchanged. In the
production layout, merging the head with its large successor produces an output
large enough for the existing promotion logic to advance the cumulative point
naturally.
- Emit a rate-limited INFO log for cloud `CUMULATIVE_NO_SUITABLE_VERSION`
submission failures, while preserving the existing debug-warning and
`BE_NO_SUITABLE_VERSION` behavior.
This is a follow-up to #59268 and is independent of #64954.
#### Testing
- `sh run-be-ut.sh --run --filter='*CumulativeCompaction*'`: 74/74 passed on
the final branch.
- Covered a production-shaped non-overlapping head followed by oversized
overlapping rowsets and empty/small rowsets; selection returns the head plus
its direct successor and the merged output advances the cumulative point from
13 to 58.
- Covered the negative branch where trimming really removes successors but
the remaining head is overlapping, so no restoration occurs.
- Existing cloud empty-rowset `skip_trim` coverage remains green.
- Temporarily overlaid the two policy changes from #64954 (removing the
promotion-size early return and guarding level trim with `size() > 1`), then
reran the same suite: 74/74 passed. The overlay was removed and is not part of
this PR.
- `build-support/clang-format.sh`, `build-support/check-format.sh`, and `git
diff --check` passed.
### Release note
Prevent cumulative compaction from repeatedly selecting an unmergeable
single non-overlapping rowset after max-score trimming, and add rate-limited
visibility for repeated no-suitable-version retries.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Max-score trimming restores one successor only when needed to
keep cumulative compaction input mergeable; repeated cloud no-suitable-version
retries are now rate-limited at INFO level.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
