Damans227 opened a new issue, #14485: URL: https://github.com/apache/cloudstack/issues/14485
### problem With `kvm.incremental.snapshot=true` and `snapshot.backup.to.secondary=false`, createVolume or createTemplate from an incremental snapshot fails randomly with: ``` errorcode 530 Failed to copy snapshot: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback ``` The agent log only says `Failed to backup <uuid> ... to <secondary>/snapshots/<account>/<volume>`. The real error is hidden because managesnapshot.sh sends qemu-img's output to /dev/null. I traced it on the host: ``` qemu-img convert ... : Could not create '<secondary>/snapshots/2/3/<uuid>': No such file or directory ``` What happens: 1. Both calls copy the snapshot into `snapshots/<account>/<volume>` on secondary. After the call, the SSVM deletes the copy and also removes the folder if it's empty ([NfsSecondaryStorageResource.java#L2116-L2120](https://github.com/apache/cloudstack/blob/0a5bf30af32bdea5a209f3f993cbd6a43301d0f9/services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java#L2116-L2120)). 2. On the next copy, `backup_snapshot()` only creates the folder if `[ ! -d $destPath ]` ([managesnapshot.sh#L237-L245](https://github.com/apache/cloudstack/blob/0a5bf30af32bdea5a209f3f993cbd6a43301d0f9/scripts/storage/qcow2/managesnapshot.sh#L237-L245)). 3. If the host has the same secondary export mounted somewhere else (configdrive, template), the NFS client can still see the folder the SSVM just removed. mkdir is skipped and qemu-img fails ([managesnapshot.sh#L285](https://github.com/apache/cloudstack/blob/0a5bf30af32bdea5a209f3f993cbd6a43301d0f9/scripts/storage/qcow2/managesnapshot.sh#L285)). Every failed copy I traced had no mkdir before qemu-img. Every good one had it. [DRAG HERE: host-trace.log] ### versions - CloudStack 4.22.1.1, same code on main (0a5bf30) - KVM: Ubuntu 26.04, QEMU 10.2.1, libvirt 12.0.0 - NFS 4.2 primary and secondary (Linux NFS server) ### The steps to reproduce the bug 1. KVM zone with NFS secondary. Set `kvm.incremental.snapshot=true` and `snapshot.backup.to.secondary=false`, restart the management server. 2. On a KVM host, keep a subfolder of the same secondary export mounted, like a configdrive mount does: `mount -t nfs -o vers=4.2 <secstore>:/<export>/configdrive /mnt/keep` 3. Deploy a VM on that host and take a full snapshot of its ROOT volume. 4. Loop: createSnapshot (incremental), then createVolume from it, then createTemplate from it. Without step 2 I got 0 failures in 8 rounds. With it, 6 of 12 rounds failed, all with the error above. [DRAG HERE: api-results.txt] [DRAG HERE: mgmt-failed-jobs.log] Same failure shows in the UI ("Failed to create volume"): [DRAG HERE: ui-create-volume-failed.png] [DRAG HERE: ui-repro-video.zip] ### What to do about it? - Run `mkdir -p "$destPath"` every time in `backup_snapshot()`. It's safe if the folder exists. - Keep qemu-img's error in the message sent back, so this shows up in the logs. - Maybe don't remove `snapshots/<account>/<volume>` when the temp copy is deleted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
