yeah, I like that idea Mike.
Also, I want to be clear that breaking out the backup to secondary into a separate thread is a great feature, so kudos to those that developed it. That along with the ability to turn it off completely makes for a nice overall improvement to snapshots. - Si ________________________________ From: Tutkowski, Mike <mike.tutkow...@netapp.com> Sent: Saturday, May 19, 2018 8:40 PM To: dev@cloudstack.apache.org Subject: Re: Snapshots only on Primary Storage feature Perhaps instead of renaming the setting, we can note in its description the hypervisors it currently pertains to. > On May 19, 2018, at 7:03 PM, Glen Baars <g...@onsitecomputers.com.au> wrote: > > Based on the responses, I think it is a worthy feature to be retained. Maybe > the following changes? > > Rename the setting to something like kvmxen.snapshot.primaryonly ( I have no > idea of the naming scheme that Cloudstack uses ) > Ensure the code for vmware snapshots does not get impacted by the setting > Record in the DB that the snapshot is only on the primary storage > When the create template or download template features are used, use the > primary storage as the source. > > Kind regards, > Glen Baars > > -----Original Message----- > From: Will Stevens <wstev...@cloudops.com> > Sent: Saturday, 19 May 2018 12:57 PM > To: dev@cloudstack.apache.org > Subject: Re: Snapshots only on Primary Storage feature > > I think reverting the change in 4.11.1 is probably a good idea. > > *Will Stevens* > Chief Technology Officer > c 514.826.0190 > > <https://goo.gl/NYZ8KK> > > > On Fri, May 18, 2018 at 2:52 PM ilya musayev <ilya.mailing.li...@gmail.com> > wrote: > >> Perhaps bring it back into 4.11.1? >> >> On Fri, May 18, 2018 at 9:28 AM Suresh Kumar Anaparti < >> sureshkumar.anapa...@gmail.com> wrote: >> >>> Si / Will, >>> >>> That is just FYI, if anyone uses VMware with that flag set to false. >>> I'm neither against the feature nor telling to rip that out. >>> >>> You are correct, the PR 2081 supports KVM and Xen as the volume >>> snapshots are directly supported on them and backup operation is not >>> tightly >> coupled >>> with the create operation. >>> >>> -Suresh >>> >>> On Fri, May 18, 2018 at 7:38 PM, Simon Weller >>> <swel...@ena.com.invalid> >>> wrote: >>> >>>> There are plenty of features in ACS that are particular to a >>>> certain hypervisor (or hypervisor set), including VMware specific items. >>>> >>>> It was never claimed this feature worked across all hypervisors. >>>> In addition to that, the default was to leave the existing >>>> functionality exactly the way it was originally implemented and if >>>> a user wished to change the functionality they could via a global config >>>> variable. >>>> >>>> Your original spec for PR 2081 in confluence states that the PR >>>> was targeted towards KVM and Xen, so I'm confused as to why VMware >>>> is even being mentioned here. >>>> >>>> >>>> This is a major feature regression that a number of >> organizations/service >>>> providers are relying on and it wasn't called out when the PR was >>> submitted. >>>> >>>> >>>> ________________________________ >>>> From: Will Stevens <wstev...@cloudops.com> >>>> Sent: Friday, May 18, 2018 6:12 AM >>>> To: dev@cloudstack.apache.org >>>> Subject: Re: Snapshots only on Primary Storage feature >>>> >>>> Just because it does not work for VMware should not a reason to >>>> rip out >>> the >>>> functionality for other hypervisors where it is being used though. >>>> >>>> I know we also have the requirement that snapshots are not >> automatically >>>> replicated to secondary storage, so this feature is useful to us. >>>> >>>> I don't understand the rational for removing the feature just >>>> because >> it >>>> does not work on VMware. >>>> >>>> On Fri, May 18, 2018, 6:27 AM Suresh Kumar Anaparti, < >>>> sureshkumar.anapa...@gmail.com> wrote: >>>> >>>>> Si, >>>>> >>>>> The PR# 1697 with the global setting >>>>> *snapshot.backup.rightafter** - >>>>> false* doesn't >>>>> work for VMware as create snapshot never takes a snapshot in >>>>> Primary >>>> pool, >>>>> it just returns the snapshot uuid. The backup snapshot does the >>> complete >>>>> job - creates a VM snapshot with the uuid, extracts and exports >>>>> the >>>> target >>>>> volume to secondary. On demand backup snapshot doesn't work as >>>>> there >> is >>>> no >>>>> snapshot in primary. Also, there'll be only one entry with >>>>> Primary >>> store >>>>> role in snapshot_store_ref, which is the latest snapshot taken >>>>> for >> that >>>>> volume. >>>>> >>>>> -Suresh >>>>> >>>>> On Fri, May 18, 2018 at 1:03 AM, Simon Weller >> <swel...@ena.com.invalid >>>> >>>>> wrote: >>>>> >>>>>> The whole point of the original PR was to optionally disable >>>>>> this functionality. >>>>>> >>>>>> We don't expose views of the backup state to our customers (we >>>>>> have >>> our >>>>>> own customer interfaces) and it's a large waste of space for >>>>>> us to >> be >>>>>> backing up tons of VM images when we have a solid primary >>>>>> storage infrastructure that already has lots of resiliency. >>>>>> >>>>>> >>>>>> I guess we're going to have to revisit this again before we >>>>>> can >>>> consider >>>>>> rebasing on 4.11. >>>>>> >>>>>> ________________________________ >>>>>> From: Suresh Kumar Anaparti <sureshkumar.anapa...@gmail.com> >>>>>> Sent: Thursday, May 17, 2018 2:21 PM >>>>>> To: dev >>>>>> Subject: Re: Snapshots only on Primary Storage feature >>>>>> >>>>>> Hi Si, >>>>>> >>>>>> No. not possible to disable the backup to secondary. It copies >>>>>> the >>>> volume >>>>>> snapshot to secondary in a background thread using asyncBackup >> param >>>> (set >>>>>> to true) and allows other operations during that time. >>>>>> >>>>>> I understand that the backup was on demand when any operations >>>>>> are performed on the snapshot. But, backup during that time >>>>>> may take considerable time (depending on the snapshot size and >>>>>> the network bandwidth), which can result in the job timeout >>>>>> and the User may >>> assume >>>>>> that it is already Backed up based on its state, unless it is >>>> documented. >>>>>> >>>>>> -Suresh >>>>>> >>>>>> On Fri, May 18, 2018 at 12:23 AM, Simon Weller >>> <swel...@ena.com.invalid >>>>> >>>>>> wrote: >>>>>> >>>>>>> Suresh, >>>>>>> >>>>>>> >>>>>>> With this new merged PR, is it possible to disable the >>>>>>> backup to secondary completely? I can't tell from the >>>>>>> reference spec and >>> we're >>>>> not >>>>>> on >>>>>>> a 4.10/4.11 base yet. >>>>>>> >>>>>>> For the record, in the instances where a volume or template >>>>>>> from >>>>> snapshot >>>>>>> was required, the backup image was copied on demand to secondary. >>>>>>> >>>>>>> In an ideal world, secondary storage wouldn't even be >>>>>>> involved in >>>> most >>>>> of >>>>>>> these options, instead using the native clone features of >>>>>>> the >>>>> underlying >>>>>>> storage. >>>>>>> >>>>>>> >>>>>>> - Si >>>>>>> >>>>>>> ________________________________ >>>>>>> From: Suresh Kumar Anaparti <sureshkumar.anapa...@gmail.com> >>>>>>> Sent: Thursday, May 17, 2018 1:37 PM >>>>>>> To: dev@cloudstack.apache.org >>>>>>> Cc: Nathan Johnson >>>>>>> Subject: Re: Snapshots only on Primary Storage feature >>>>>>> >>>>>>> Hi Glen / Si, >>>>>>> >>>>>>> In PR# 1697, the global setting *snapshot.backup.rightafter* >>>>>>> if >> set >>>> to >>>>>>> true, it'll be the default behaviour and snapshot is copied >>>>>>> to >> the >>>>>>> secondary storage. If set to false, then the snapshot state >>>> transitions >>>>>> are >>>>>>> mocked and Snapshot would be in BackedUp state even though >>>>>>> it is >>> not >>>>>> really >>>>>>> in Secondary storage, which doesn't make sense. Also, that >>>>>>> will >>>> enable >>>>> to >>>>>>> create a volume or template from the snapshot, which will >> obviously >>>>> fail. >>>>>>> >>>>>>> This behavior was changed with the PR >>>>>>> https://github.com/apache/cloudstack/pull/2081. There is a >>>>>>> clear separation of create and backup volume snapshot >>>>>>> operations. The global >> setting >>>>>>> *snapshot.backup.rightafter* has been removed in PR# 2081. >>>>>>> >>>>>>> -Suresh >>>>>>> >>>>>>> On Thu, May 17, 2018 at 8:40 PM, Simon Weller >>>> <swel...@ena.com.invalid >>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Glen, >>>>>>>> >>>>>>>> >>>>>>>> This feature was implemented in 4.9 by my colleague Nathan >>> Johnson. >>>>>> You >>>>>>>> enable it by changing the global setting >>>> snapshot.backup.rightafter >>>>> to >>>>>>>> false. >>>>>>>> >>>>>>>> >>>>>>>> The PR is reference here: https://github.com/apache/ >>>>>> cloudstack/pull/1697 >>>>>>>> >>>>>>>> >>>>>>>> We have the exact same use case as you, as we also use Ceph. >>>>>>>> >>>>>>>> >>>>>>>> - Si >>>>>>>> >>>>>>>> >>>>>>>> ________________________________ >>>>>>>> From: Glen Baars <g...@onsitecomputers.com.au> >>>>>>>> Sent: Thursday, May 17, 2018 9:46 AM >>>>>>>> To: dev@cloudstack.apache.org >>>>>>>> Subject: Snapshots only on Primary Storage feature >>>>>>>> >>>>>>>> >>>>>>>> Hello Devs, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I have been thinking about a feature request and want to >>>>>>>> see >> what >>>>>> people >>>>>>>> think about the use case. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We use KVM + Ceph RBD as storage. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Currently, when a client takes a snapshot, Cloudstack >>>>>>>> takes a >>> Ceph >>>>>>>> snapshot and then uses qemu-img to export to secondary storage. >>>> This >>>>>>>> creates a full backup of the server. Clients want to use >>>>>>>> this >> as >>> a >>>>>> daily >>>>>>>> snapshot and it isn’t feasible due to the space requirements. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We would like create the snapshot only on primary storage. >>>>>>>> It >> is >>>>>>>> replicated offsite and fault tolerant. I can see that the >>> download >>>>>>> snapshot >>>>>>>> and create template features may be an issue. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I have seen the below features in the recent releases and >>> wondered >>>> if >>>>>>> this >>>>>>>> was the direction that the development was going. >>>>>>>> >>>>>>>> Separation of volume snapshot creation on primary storage >>>>>>>> and >>>> backing >>>>>>>> operation on secondary storage. >>>>>>>> >>>>>>>> Bypass secondary storage template copy/transfer for KVM. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> >>>>>>>> Glen Baars >>>>>>>> >>>>>>>> BackOnline Manager >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> T 1300 733 328 / +61 8 6102 3276 >>>>>>>> >>>>>>>> NZ +64 9280 3561 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> www.timg.com<http://www.timg.com/> >>>>>>>> >>>>>>>> [http://images.dbonline.com.au/images/fb.png] Facebook< >>>>>>>> https://www.facebook.com/TheInformationManagementGroup> [ >>>>>>>> http://images.dbonline.com.au/images/li.png] LinkedIn< >>>>>>> http://www.linkedin. >>>>>>>> com/company/the-information-management-group?trk=hb_tab_ >>>>>>> compy_id_2724246> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Watch a short video about what we >>>>>>>> do!<https://www.youtube.com/ watch?v=scLGLwSIFQc> >>>>>>>> >>>>>>>> [http://images.dbonline.com.au/images/timgv3.jpg]<https:// >>>>>> goo.gl/eAHLO7> >>>>>>>> >>>>>>>> This e-mail may contain confidential and/or privileged >>>> information.If >>>>>> you >>>>>>>> are not the intended recipient (or have received this >>>>>>>> e-mail in >>>>> error) >>>>>>>> please notify the sender immediately and destroy this e-mail. >> Any >>>>>>>> unauthorized copying, disclosure or distribution of the >> material >>> in >>>>>> this >>>>>>>> e-mail is strictly forbidden. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This e-mail is intended solely for the benefit of the >>> addressee(s) >>>>> and >>>>>>> any >>>>>>>> other named recipient. It is confidential and may contain >> legally >>>>>>>> privileged or confidential information. If you are not the >>>> recipient, >>>>>> any >>>>>>>> use, distribution, disclosure or copying of this e-mail is >>>>> prohibited. >>>>>>> The >>>>>>>> confidentiality and legal privilege attached to this >>> communication >>>> is >>>>>> not >>>>>>>> waived or lost by reason of the mistaken transmission or >> delivery >>>> to >>>>>> you. >>>>>>>> If you have received this e-mail in error, please notify >>>>>>>> us >>>>>> immediately. >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately.
