calvix opened a new pull request, #13835:
URL: https://github.com/apache/cloudstack/pull/13835
takeRbdVolumeSnapshotOfStoppedVm() called image.snapCreate(snapshotName)
twice. The first call creates the RBD snapshot, the second one always
throws RbdException ("Failed to create snapshot <uuid>") because the
snapshot already exists.
The duplicate happened in a merge artifact: 30d3066 ("Merge branch '4.20'
into
4.22")
https://github.com/apache/cloudstack/commit/30d306622a90ac43f2a6c35ee999110ad1bc5194#diff-8a24835eeac038da0df1229615ce6ce564f2e49273abb55f4c9a90dedd29c222L2318-R2333
Because there was no finally block, that exception skipped rbd.close(image)
and r.ioCtxDestroy(io), so the agent kept the image open and held its RBD
exclusive-lock indefinitely. The exception is only logged, so the snapshot
job still reported success and the fault stayed invisible.
Consequences observed on a KVM + Ceph/RBD cluster:
- revertSnapshot fails with "com.ceph.rbd.RbdException: Failed to rollback
snapshot <uuid>". librbd returns EROFS because a live peer holds the
exclusive-lock; 'rbd snap rollback' only succeeds once that client dies
and librbd can break the lock, which makes the failure look intermittent.
- getRbdSnapshotSize() is never reached, so every snapshot is reported with
physical size 0 when snapshot.backup.to.secondary is false.
- The leaked watchers keep the image busy, so 'rbd rm' fails and the volume
cannot be expunged - it stays stuck in state Destroy.
Remove the duplicated call and move the image/IO-context cleanup into a
finally block so the lock is released even if the snapshot itself fails.###
Description
This PR...
<!--- Describe your changes in DETAIL - And how has behaviour functionally
changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to
reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be
closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!---
*******************************************************************************
-->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE
DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!---
*******************************************************************************
-->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to
-->
**Manual testing** on a KVM + Ceph/RBD cluster (CloudStack 4.22.1.0, Ceph
20.2.2, single RBD pool, tested with `snapshot.backup.to.secondary` both `true`
and `false`):
- `createSnapshot` on a DATADISK, stop the VM, `revertSnapshot` →
`RbdException: Failed to rollback snapshot`.
- The RBD snapshot itself exists and is correctly named, so the rollback
target was never the problem — `rbd snap ls` returned exactly the name passed
to `snapRollBack`.
- Correlated success/failure with the lock owner: when the lock holder still
appears in `rbd status` (live client) the revert fails; when it does not (dead
client, librbd breaks the lock), it succeeds.
- Confirmed the mechanism by clearing the lock by hand:
- After clearing the lock, CloudStack `revertSnapshot` API job also
returns success.
```
# rbd lock rm ....
```
<!-- Please read the
[CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
