stevenpall opened a new pull request, #65612:
URL: https://github.com/apache/doris/pull/65612
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
When the cloud recycler processes a PREPARED rowset (the output of a
compaction job that died mid-flight, e.g. because its BE crashed),
`InstanceRecycler::abort_job_for_related_rowset` must first abort the recorded
compaction job. Two failure modes make that abort permanently impossible, so
the rowset is never recycled and the oldest such entry pins the whole
`recycle_rowset` queue watermark forever:
1. **Expired job cannot be aborted.** `process_compaction_job` rejects every
action on an expired compaction job with `JOB_EXPIRED` before the action
dispatch (the existing `// FIXME: Just remove or notify to abort?` marks
exactly this). The recycler needs the ABORT to succeed, but the job expired
long ago — circular, and it never self-heals. An expired job can never COMMIT
(rejected by this same check), so allowing ABORT to erase it is strictly
cleanup and loses nothing.
2. **Job-not-found treated as fatal.** If the recorded job no longer matches
the rowset's job when `_finish_tablet_job` re-resolves it (job id already
removed, or the job key referenced by the job's embedded tablet idx no longer
exists → `INVALID_ARGUMENT` "job not found"), the recycler returns -1 and
retries forever. For the recycler's purposes "nothing to abort" means it is
safe to proceed with recycling, mirroring the existing early-return when the
job key is absent at the recycler's own read ("job not exists ... return 0").
Observed on a production compute-storage-decoupled cluster (4.1): a BE OOM
crash loop killed several BASE compactions mid-flight; 7 PREPARED rowsets then
failed every recycler pass for 15 days with:
```
recycler.cpp:1848] failed to abort job, ... code=5000 msg=... expired
compaction job, tablet_id=... job={...}
recycler.cpp:2058] failed to abort txn or job for related rowset, ...
```
and
```
recycler.cpp:1848] failed to abort job, ... code=1001 msg=... job not found,
... err=KeyNotFound
recycler.cpp:2058] failed to abort txn or job for related rowset, ...
```
pinning the `recycle_rowset` queue watermark at the crash date while every
other queue stayed current.
Fix:
- `meta_service_job.cpp`: skip the expiration short-circuit for `ABORT`
actions, so aborting an expired compaction job erases it as usual (COMMIT and
LEASE behavior unchanged).
- `recycler.cpp`: in `abort_job_for_related_rowset`, treat `JOB_EXPIRED`
(older meta-service without the change above) and `INVALID_ARGUMENT` (job/idx
mismatch, job already gone) as "nothing left to abort" and proceed with
recycling instead of failing the rowset forever.
### Release note
Fix cloud recycler permanently failing to recycle prepared rowsets whose
compaction job has expired or no longer exists, which pinned the recycle_rowset
queue watermark.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Reproduced on a production 4.1 compute-storage-decoupled cluster:
7 PREPARED rowsets from BE-crash-killed BASE compactions failed
`abort_job_for_related_rowset` on every recycler pass for 15 days (both
`JOB_EXPIRED` and `INVALID_ARGUMENT`/KeyNotFound flavors), freezing the
`recycle_rowset` watermark at the crash timestamp while other queues stayed
current. With the fix the abort path resolves and the rowsets recycle.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. ABORT of an expired compaction job now succeeds (erases the
recorded job) instead of returning JOB_EXPIRED; the recycler proceeds to
recycle a prepared rowset when there is no abortable job left instead of
retrying forever.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
