May I suggest another permanent fix (sounds like a joke, but I'm actually serious) - move away from Primary Storage whose behavior is to copy snapshoted data to Secondary Storage - to the Primary Storage that doesn't copy content to Secondary Storage -i.e. SolidFire :)
Imagine having 500GB disk customer, who configured hourly snapshots with CEPH/NFS.... this is the reason we at first disabled hourly snapshots in GUI :) We also sometimes hit this problem, all CEPH and NFS volumes (ACS 4.5, ACS 4.8) and this is due to sometimes also concurrent snaps on same volume or VM, this is more of ACS logic problem, not underlying infra problem - i.e. volume owner configured both daily and weekly/monthly snapshot to start at identical time (same HH:MM), so one of them is stuck in Allocated state etc (and BTW this was on SolidFire, where snapshots are instant, stays on SolidFire, and takes i.e. 3 seconds to make proper snap to BackedUp state - even here we had issues, agani pure ACS/user logic root cause) Best Andrija On 31 August 2017 at 10:15, Ivan Kudryavtsev <[email protected]> wrote: > Hi, Dag. > > You're really right about that, but I mean that If the cloud is running and > you don't have developer capacity to investigate, the long term solution is > that works for you right now. It's really workaround and I don't pretend > arguing with that. > > 2017-08-31 15:08 GMT+07:00 Dag Sonstebo <[email protected]>: > > > Hi Ivan, > > > > Thanks for the guide, good workaround for someone hitting the same issue. > > I must respectfully disagree with your sentiment though. As you have > > mentioned in your article issues like this tend to have a root cause – > > network outage, hypervisor issues etc. – and these issues vary from > > environment to environment and should be investigated on a case by case > > basis. Your suggested long term solution is just a workaround – error > > trapping and subsequent housekeeping should be improved in the CloudStack > > code, so the best thing to do is investigate, find root cause and log a > > Jira ticket such that it can be improved going forward. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > On 31/08/2017, 06:33, "Ivan Kudryavtsev" <[email protected]> > wrote: > > > > Hello, community. > > > > Today we (again) met snapshots stuck in exotic states and I decided > to > > create small guide how to fix it. AFAIK, somebody tried to manage KB > in > > GitHub about useful tools, cases, etc. It might be that small article > > can > > be included. I wrote it because short googling didn't give positive > > results > > about the problem. Also, all comments are welcome. > > > > https://bitworks.software/blog/en/2017-08-31-cloudstack- > > removing-snapshots-stuck-in-allocated-backingup-states.html > > > > Also, the same for Russian-speaking people: > > > > https://bitworks.software/blog/2017-08-31-cloudstack- > > removing-snapshots-stuck-in-allocated-backingup-states.html > > > > -- > > With best regards, Ivan Kudryavtsev > > Bitworks Software, Ltd. > > Cell: +7-923-414-1515 > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > [email protected] > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > > -- > With best regards, Ivan Kudryavtsev > Bitworks Software, Ltd. > Cell: +7-923-414-1515 > WWW: http://bitworks.software/ <http://bw-sw.com/> > -- Andrija Panić
