bobhan1 opened a new pull request, #68436:
URL: https://github.com/apache/doris/pull/68436
### What problem does this PR solve?
Cloud schema change can allow cumulative compaction on a `TABLET_NOTREADY`
tablet before its delete bitmap is complete. Post-compaction prefill then
caches empty or incomplete aggregate bitmaps at the tablet's maximum version.
Schema change later installs the completed bitmap and marks the tablet
`TABLET_RUNNING`, but the cache keys are unchanged. Queries can reuse the stale
bitmaps and return overwritten rows.
Require `TABLET_RUNNING` under the existing tablet header read lock before
either post-compaction prefill mode runs. Normal prefill on running tablets
remains enabled.
Add a parameterized BE test that keeps the same rowsets and read version
while installing the schema-change bitmap and transitioning the tablet to
RUNNING. It covers all four prefill switch combinations, output and non-output
rowsets, and non-contiguous segment IDs. Each test instance uses unique rowset
IDs to isolate the global aggregate cache.
### Release note
Fix duplicate rows returned by Unique Key Merge-on-Write queries when schema
change overlaps compaction.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
Validation:
- Before the fix: three combinations with prefill enabled fail with stale
empty bitmaps; disabling both switches passes.
- After the fix: 56 ASAN BE unit tests pass, including all four new
parameter combinations, `CloudTabletDeleteBitmapTest`, `TabletMetaTest`, and
`CloudCompactionTest`.
- Command: `./run-be-ut.sh --run
--filter='PrefillModes/CloudTabletDeleteBitmapPrefillTest.*:CloudTabletDeleteBitmapTest.*:TabletMetaTest.*:CloudCompactionTest.*'
-j100`
- Changed C++ files pass clang-format 16, build hygiene checks, and `git
diff --check`.
- The existing Docker regression cases
`test_schema_change_with_compaction9` and
`test_schema_change_with_compaction10` cover the original interleaving but were
not run locally. Final-change clang-tidy was not run.
- Behavior changed:
- [ ] No.
- [x] Yes. Post-compaction aggregate delete bitmap cache prefill is
skipped until the tablet is RUNNING.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
