Hi Guys, I have already logged this as a big under reference: 4797
Ed On Thu, 17 Jun 2021 at 06:37, Suresh Anaparti <[email protected]> wrote: > Hi Andrei, > > Can you check if the storage garbage collector is enabled or not in your > env (specified using the global setting 'storage.cleanup.enabled'). If it > is enabled, check the interval & delay setting: 'storage.cleanup.interval' > and 'storage.cleanup.delay', and see the logs to confirm cleanup is > performed or not. > > Also, check the snapshot status / state in snapshots & snapshot_store_ref > tables for the snapshots that are not deleted during the cleanup. Is > 'removed' timestamp set for them in snapshots table? > > Regards, > Suresh > > On 16/06/21, 9:46 PM, "Andrei Mikhailovsky" <[email protected]> > wrote: > > Hello, > > I've done some more investigation and indeed, the snapshots were not > taken because the secondary storage was over 90% used. I have started > cleaning some of the older volumes and noticed another problem. After > removing snapshots, they do not seem to be removed from the secondary > storage. I've removed all snapshots over 24 hours ago and it looks like > the disk space hasn't been freed up at all. > > Looks like there are issues with snapshotting function after all. > > Andrei > > > > > > > ----- Original Message ----- > > From: "Harikrishna Patnala" <[email protected]> > > To: "users" <[email protected]> > > Sent: Tuesday, 8 June, 2021 03:33:57 > > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > > > Hi Andrei, > > > > Can you check the following things and let us know? > > > > > > 1. Can you try creating a new volume and then create snapshot of > that, to check > > if this an issue with old entries > > 2. For the snapshots which are failing can you check if you are > seeing any > > error messages like this "Can't find an image storage in zone with > less than". > > This is to check if secondary storage free space check failed. > > 3. For the snapshots which are failing and if it is delta snapshot > can you > > check if its parent's snapshot entry exists in "snapshot_store_ref" > table with > > 'parent_snapshot_id' of the current snapshot with 'store_role' > "Image". This is > > to find the secondary storage where the parent snapshot backup is > located. > > > > Regards, > > Harikrishna > > ________________________________ > > From: Andrei Mikhailovsky <[email protected]> > > Sent: Monday, June 7, 2021 7:00 PM > > To: users <[email protected]> > > Subject: Snapshots are not working after upgrading to 4.15.0 > > > > Hello everyone, > > > > I am having an issue with volume snapshots since I've upgraded to > 4.15.0. None > > of the volumes are being snapshotted regardless if the snapshot is > initiated > > manually or from the schedule. The strange thing is that if I > manually take the > > snapshot, the GUI shows Success status, but the Storage>Snapshots > show an Error > > status. Here is what I see in the management server logs: > > > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) > (logid:be34ce01) Done > > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 > > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) > (logid:be34ce01) Remove > > job-86143 from job monitoring > > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy > snapshot > > com.cloud.utils.exception.CloudRuntimeException: can not find an > image stores > > at > > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) > > at > > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > > at > > > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > > at > > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of > snapshot > > failed, for snapshot with ID 53531, left with 2 more attempts > > > > > > I've checked and the Secondary storage is configured and visible in > the GUI. I > > can also mount it manually from the management server and a couple > of host > > servers that I've tested. In addition, I can successfully upload an > ISO image > > and that registers just fine and I can create new VMs using the > newly uploaded > > ISO image. > > > > I've had no such problems with 4.13.x ACS, so the issue seems to > have been > > introduced after doing the upgrade to 4.15.0. > > > > Could you please let me know how do I fix the issue? > > > > Cheers > > > > andrei > >
