Hi Andrei, Can you check if the storage garbage collector is enabled or not in your env (specified using the global setting 'storage.cleanup.enabled'). If it is enabled, check the interval & delay setting: 'storage.cleanup.interval' and 'storage.cleanup.delay', and see the logs to confirm cleanup is performed or not.
Also, check the snapshot status / state in snapshots & snapshot_store_ref tables for the snapshots that are not deleted during the cleanup. Is 'removed' timestamp set for them in snapshots table? Regards, Suresh On 16/06/21, 9:46 PM, "Andrei Mikhailovsky" <[email protected]> wrote: Hello, I've done some more investigation and indeed, the snapshots were not taken because the secondary storage was over 90% used. I have started cleaning some of the older volumes and noticed another problem. After removing snapshots, they do not seem to be removed from the secondary storage. I've removed all snapshots over 24 hours ago and it looks like the disk space hasn't been freed up at all. Looks like there are issues with snapshotting function after all. Andrei ----- Original Message ----- > From: "Harikrishna Patnala" <[email protected]> > To: "users" <[email protected]> > Sent: Tuesday, 8 June, 2021 03:33:57 > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > Hi Andrei, > > Can you check the following things and let us know? > > > 1. Can you try creating a new volume and then create snapshot of that, to check > if this an issue with old entries > 2. For the snapshots which are failing can you check if you are seeing any > error messages like this "Can't find an image storage in zone with less than". > This is to check if secondary storage free space check failed. > 3. For the snapshots which are failing and if it is delta snapshot can you > check if its parent's snapshot entry exists in "snapshot_store_ref" table with > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". This is > to find the secondary storage where the parent snapshot backup is located. > > Regards, > Harikrishna > ________________________________ > From: Andrei Mikhailovsky <[email protected]> > Sent: Monday, June 7, 2021 7:00 PM > To: users <[email protected]> > Subject: Snapshots are not working after upgrading to 4.15.0 > > Hello everyone, > > I am having an issue with volume snapshots since I've upgraded to 4.15.0. None > of the volumes are being snapshotted regardless if the snapshot is initiated > manually or from the schedule. The strange thing is that if I manually take the > snapshot, the GUI shows Success status, but the Storage>Snapshots show an Error > status. Here is what I see in the management server logs: > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Done > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Remove > job-86143 from job monitoring > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy snapshot > com.cloud.utils.exception.CloudRuntimeException: can not find an image stores > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) > at > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > at > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of snapshot > failed, for snapshot with ID 53531, left with 2 more attempts > > > I've checked and the Secondary storage is configured and visible in the GUI. I > can also mount it manually from the management server and a couple of host > servers that I've tested. In addition, I can successfully upload an ISO image > and that registers just fine and I can create new VMs using the newly uploaded > ISO image. > > I've had no such problems with 4.13.x ACS, so the issue seems to have been > introduced after doing the upgrade to 4.15.0. > > Could you please let me know how do I fix the issue? > > Cheers > > andrei
