Yukang-Lian opened a new pull request, #69134:
URL: https://github.com/apache/doris/pull/69134
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
In cloud mode, a schema change that has to sort data (e.g. adding a rollup
or a sync materialized view whose key columns are in a different order) writes
the intermediate rowsets of internal sorting to a local path. These rowsets
have no storage resource, so `CloudRowsetWriter::init` puts them in the root of
a file cache path as `<rowset_id>_<segment_id>.dat`. Every input rowset with
data produces at least one intermediate rowset, so each such schema change
writes a full copy of the converted data there.
Only the local mode subclass `VLocalSchemaChangeWithSorting` removes the
intermediate rowsets. Cloud mode uses `VBaseSchemaChangeWithSorting` directly,
which never removes them, whether the schema change succeeds or fails. These
files are not part of the file cache: cache eviction and `clear_file_cache` API
do not see them, and the cache loader skips regular files in the cache root on
restart. They pile up on the cache disk until it is full, the file cache then
evicts its own data and cannot admit new blocks, and reads fall back to remote
storage. We observed hundreds of GB of such files on a cluster which created
sync materialized views repeatedly.
This PR:
- Adds `VCloudSchemaChangeWithSorting`, used by cloud schema change, which
removes the intermediate rowsets after each input rowset is processed, whether
it succeeds or fails (like `VLocalSchemaChangeWithSorting` does in local mode).
- Removes the local rowset files left in file cache paths by previous BE
processes (e.g. BE crashed during schema change) when `CloudStorageEngine`
opens. A rowset id embeds the backend uid of the process that generated it,
which is regenerated on every start, so only files from previous processes are
removed and files written by the current process are never touched. Only
regular files whose names match the local rowset file format are considered.
- Adds a debug point
`VBaseSchemaChangeWithSorting._inner_process.before_external_sorting` to inject
a failure in tests.
### Release note
Fix the intermediate rowsets of schema change with sorting being left in the
file cache path in cloud mode, which could fill up the cache disk.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit tests (`CloudSchemaChangeSortingTest`) run a real sorting schema
change in cloud mode and check the file cache root after success and after an
injected failure, and cover the startup cleanup of stale rowset files. Without
the cleanup, the success and failure cases fail with the intermediate rowsets
left in the cache root.
The docker regression test `test_cloud_sc_sorting_tmp_rowset_cleanup`
covers a rollup that reorders keys, a failed rollup, and BE restart with stale
rowset files in the cache root. On the code before this PR it fails at the
first step (`expected: <0> but was: <5>`, one leftover intermediate rowset per
loaded rowset).
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
In cloud mode, intermediate rowsets of schema change with sorting are
removed after use, and stale ones in file cache paths are removed when BE
starts.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
