So, right now in 4.6, 4.7, and 4.8, the behavior for a managed storage
volume snapshot is for the data to remain on the SAN (not secondary
storage).
It was simply designed as a space-efficient and fast alternative to copying
data to NFS (secondary storage).
What we need to do is somehow maintain that original behavior, but augment
it with the option of backing up the data to secondary storage (and, if
doing that, then delete the SAN snapshot).
On Mon, Feb 8, 2016 at 11:55 AM, Ian Rae <i...@cloudops.com> wrote:
> I would hope that default behaviour in CloudStack is the traditional volume
> snapshot moved to secondary storage. A global setting to change that
> behaviour is probably ok if it is not default, but the user would want to
> in certain cases make copies of those snapshots to secondary storage in
> addition to taking the snapshot on the primary storage.
>
> On Monday, February 8, 2016, Will Stevens <wstev...@cloudops.com> wrote:
>
> > I don't think a global setting is a good option because we need both
> > functionality to be available at the same time and for different use
> cases
> > to be able to pick which they choose.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com <javascript:;>
> > > wrote:
> >
> > > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> > care
> > > where the snapshot is.
> > >
> > > If that's true, then we should perhaps control this via Global Settings
> > or
> > > something.
> > >
> > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > mike.tutkow...@solidfire.com> wrote:
> > >
> > > > It's not ideal - true, but it does allow us to be backward
> compatible.
> > > >
> > > > If you have other ideas, though, about how to maintain backward
> > > > compatibility, I'm definitely open to hear them.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> syed1.mush...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Mike,
> > > >>
> > > >> Adding a flag to createSnapshot was the first and the most obvious
> > thing
> > > >> that came to our minds. The problem that I had with this was that:
> > > >>
> > > >> 1) I feel it is exposing something to the end user that is internal
> to
> > > the
> > > >> cloud.
> > > >>
> > > >> 2) We have to follow two different ways of restore/deletion in the
> > same
> > > >> code path depending on where the Snapshot resides which I find kind
> > of a
> > > >> bad design.
> > > >>
> > > >> But if exposing a archive flag to the end user is acceptable then we
> > can
> > > >> definitely use this instead of adding the StorageSnapshot API
> > > >>
> > > >> Thanks,
> > > >> -Syed
> > > >>
> > > >>
> > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > >> mike.tutkow...@solidfire.com
> > > >> > wrote:
> > > >>
> > > >> > Hi Pierre-Luc,
> > > >> >
> > > >> > My recommendation would be this:
> > > >> >
> > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > default
> > > >> would
> > > >> > be "false" because that would be backward compatible with how 4.6
> > has
> > > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6,
> 4.7,
> > > and
> > > >> > 4.8).
> > > >> >
> > > >> > If you set archive=true, then we would perform a background
> > migration
> > > of
> > > >> > the snapshot from the SAN to the secondary storage (then delete
> the
> > > SAN
> > > >> > snapshot).
> > > >> >
> > > >> > That archive parameter would only be valid for managed storage.
> > > >> >
> > > >> > Sound reasonable?
> > > >> >
> > > >> > Also, a VM snapshot that includes disks provided by managed
> storage
> > > >> should
> > > >> > work.
> > > >> >
> > > >> > Talk to you later,
> > > >> > Mike
> > > >> >
> > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> pd...@cloudops.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Mike,
> > > >> > >
> > > >> > > In terms of API's, would you prefer introducing a parameter to
> the
> > > >> > existing
> > > >> > > VolumeSnapshot, example: extract={true|false} with a default
> > > value
> > > >> of
> > > >> > > true which would extract snapshot into the secondary storage,
> > which
> > > >> is
> > > >> > the
> > > >> > > current default behavior. Then for SAN snapshot that remain on
> the
> > > >> SAN we
> > > >> > > would just set "extract=false" ? as oppose to create a new
> > > >> > > StorageSnapshot API ?
> > > >> > >
> > > >> > >
> > > >> > > Paul,
> > > >> > >
> > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> using
> > > >> managed
> > > >> > > storage for VM having more than one Volume. For the reason that
> > > >> snapshot
> > > >> > > are performed outside of the hypervisor awareness and
> > > asynchronously.
> > > >> If
> > > >> > > someone have a way to address that, it would make thinks much
> more
> > > >> > > attractive.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <i...@cloudops.com>
> > wrote:
> > > >> > >
> > > >> > > > I think a service provider backup scenario is more likely to
> > take
> > > >> > > advantage
> > > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > > >> backups
> > > >> > > > involve access to the file system, and there is an expectation
> > > that
> > > >> > this
> > > >> > > > can be done with reasonably short time frames without
> negatively
> > > >> > > impacting
> > > >> > > > VM performance, and that the backup orchestrator can apply
> > various
> > > >> > logic
> > > >> > > > and or transformations to the data (compress, encrypt, deltas
> > > >> etc...).
> > > >> > > > While it is true that one could apply a backup process to a
> > cloud
> > > >> > > snapshot,
> > > >> > > > this would be slow and inefficient requiring the data to be
> > moved
> > > >> > several
> > > >> > > > times and there are some major bottlenecks with cloud
> snapshots.
> > > >> With a
> > > >> > > > cloud snapshot - there seems to be no reasonable expectation
> of
> > > >> being
> > > >> > > able
> > > >> > > > to do differential snapshots (I think this depends on the
> > > >> hypervisor)
> > > >> > and
> > > >> > > > if you do differential snapshots this will make file backups
> > from
> > > >> those
> > > >> > > > snapshots even more complicated to orchestrate.
> > > >> > > >
> > > >> > > > Suspect there needs to be a different thread on how to better
> > > enable
> > > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > > related
> > > >> > > > workflow so relevant to this discussion.
> > > >> > > >
> > > >> > > > Ian
> > > >> > > >
> > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > >> > > mike.tutkow...@solidfire.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > To me it sounds like number two and number three are
> different
> > > >> uses
> > > >> > for
> > > >> > > > the
> > > >> > > > > same "thing"(which is totally fine).
> > > >> > > > >
> > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > >> asynchronously, do
> > > >> > > we
> > > >> > > > > see the SSVM as performing the export?
> > > >> > > > >
> > > >> > > > > To be backwards compatible with what we have in 4.6 and
> later
> > > for
> > > >> > > volume
> > > >> > > > > snapshots for managed storage, I think it might be easier if
> > we
> > > >> pass
> > > >> > > in a
> > > >> > > > > flag that says whether or not to archive the SAN snapshot
> > > (which,
> > > >> I
> > > >> > > > think,
> > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > >> > > > >
> > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > pd...@cloudops.com
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > >
> > > >> > > > > > Hi Mike,
> > > >> > > > > >
> > > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > > >> exported
> > > >> > > into
> > > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > > >> directly
> > > >> > > > would
> > > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > > time.
> > > >> > Has
> > > >> > > > > oppose
> > > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > > >> > afterward
> > > >> > > be
> > > >> > > > > > clone into Secondary Storage asynchronously.
> > > >> > > > > >
> > > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a
> SAN
> > > >> > snapshot
> > > >> > > > > could
> > > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > > >> consumption
> > > >> > > on
> > > >> > > > > the
> > > >> > > > > > Primary Storage side.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > I see 3 definitions in our current discussion regarding
> the
> > > term
> > > >> > > > snapshot
> > > >> > > > > > (these are not official terminology but by own
> > interpretation
> > > of
> > > >> > > them):
> > > >> > > > > >
> > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition
> of
> > a
> > > >> > > > snapshot):
> > > >> > > > > > it's a volume snapshot at the storage level, point in time
> > of
> > > >> your
> > > >> > > > data.
> > > >> > > > > it
> > > >> > > > > > reside on the primary storage. Useful and efficient for
> > > software
> > > >> > side
> > > >> > > > > > incident.
> > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> cloud
> > > >> backup
> > > >> > > > aws-S3
> > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> reside
> > > on a
> > > >> > > > > different
> > > >> > > > > > storage array then the original Volume. Facilitate data
> > > >> migration
> > > >> > > > between
> > > >> > > > > > clusters and, in case of primary storage incident, Volume
> > > >> snapshots
> > > >> > > are
> > > >> > > > > not
> > > >> > > > > > impacted and can be reuse.
> > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> > also
> > > >> > > validate
> > > >> > > > > data
> > > >> > > > > > integrity, provide a storage efficient archiving method
> and
> > an
> > > >> > > > > independent
> > > >> > > > > > way to restore your data in case of an major
> infrastructure
> > > >> > disaster.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > >
> > > >> > > > > > PL
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > >> > > > > > mike.tutkow...@solidfire.com <javascript:;>
> <javascript:;>
> > > >> > > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > So, let's see if I currently follow the requirements:
> > > >> > > > > > >
> > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > >> conditionally
> > > >> > > > export
> > > >> > > > > > data
> > > >> > > > > > > to NFS. The current process of taking a snapshot on the
> > SAN
> > > is
> > > >> > > fine,
> > > >> > > > > but
> > > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > > >> > > > > > >
> > > >> > > > > > > Questions:
> > > >> > > > > > >
> > > >> > > > > > > Once the data has been exported to NFS, do we keep the
> SAN
> > > >> > snapshot
> > > >> > > > or
> > > >> > > > > > > delete it?
> > > >> > > > > > >
> > > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> > just
> > > >> copy
> > > >> > > the
> > > >> > > > > VHD
> > > >> > > > > > > from primary to secondary the way we do today for
> > > non-managed
> > > >> > (i.e.
> > > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > > >> scenario?
> > > >> > > > > Perhaps
> > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > secondary
> > > >> > > storage
> > > >> > > > > > > instead of a XenServer host?
> > > >> > > > > > >
> > > >> > > > > > > Thanks for the clarification.
> > > >> > > > > > >
> > > >> > > > > > > By the way, to me a backup is when you copy data from
> one
> > > >> storage
> > > >> > > > > system
> > > >> > > > > > to
> > > >> > > > > > > another (regardless of features, if any, to restore the
> > data
> > > >> in
> > > >> > the
> > > >> > > > > > > future). A snapshot is a point-in-time view of the data
> > of a
> > > >> > volume
> > > >> > > > and
> > > >> > > > > > > it's stored on the same storage system as the volume.
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > >> > > pd...@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > That's fun to see that discussion happening. I 100%
> > agree
> > > >> with
> > > >> > > > Paul's
> > > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> > do
> > > >> > > consider
> > > >> > > > > them
> > > >> > > > > > > as
> > > >> > > > > > > > a safety vest against Primary Storage failure, because
> > > >> failure
> > > >> > > > append
> > > >> > > > > > > :-( .
> > > >> > > > > > > >
> > > >> > > > > > > > The current proposal around snapshots that reside on
> the
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > or
> > > >> > > > > > > > snapshots that end in the Secondary Storage is not to
> > > >> address
> > > >> > > any
> > > >> > > > > kind
> > > >> > > > > > > of
> > > >> > > > > > > > backups requirement because a snapshot is not a
> backup,
> > > >> event
> > > >> > an
> > > >> > > > > > > extracted
> > > >> > > > > > > > VM snapshot.
> > > >> > > > > > > >
> > > >> > > > > > > > The main idea, and again this is for managed storage;
> > > >> > > > > > > >
> > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > > >> capability
> > > >> > > for
> > > >> > > > > > fast
> > > >> > > > > > > > response time that support rollback to previous
> > timestamp,
> > > >> > create
> > > >> > > > new
> > > >> > > > > > > > volume and maybe create template.
> > > >> > > > > > > > not required to be a new API if the work is
> already
> > > >> done, I
> > > >> > > > think
> > > >> > > > > > > this
> > > >> > > > > > > > is a different behaviors than the user expectation of
> a
> > > >> > > > > > volume-snapshot.
> > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > behavior
> > > >> that
> > > >> > > > create
> > > >> > > > > > an
> > > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> > be
> > > >> reuse
> > > >> > > to
> > > >> > > > > > > create a
> > > >> > > > > > > > new volume into another Primary Storage. This type of
> > > >> snapshot
> > > >> > > is a
> > > >> > > > > > slow
> > > >> > > > > > > > job since yes it would have to copy the full volume
> size
> > > on
> > > >> the
> > > >> > > > > > Secondary
> > > >> > > > > > > > storage.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > PL
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > >> > > > > syed1.mush...@gmail.com <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> Backup
> > > (via
> > > >> > > Volume
> > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This
> is
> > > >> > > amplified
> > > >> > > > > > > > especially
> > > >> > > > > > > > > when you have a object storage as your secondary
> > storage
> > > >> > > because
> > > >> > > > it
> > > >> > > > > > > > > requires two copies (one to an NFS staging area and
> > from
> > > >> > there
> > > >> > > to
> > > >> > > > > > > object
> > > >> > > > > > > > > storage). And not to mention that all these copies
> are
> > > >> > > consuming
> > > >> > > > > > > > hypervisor
> > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> bottleneck
> > as
> > > >> all
> > > >> > > the
> > > >> > > > > > > Network
> > > >> > > > > > > > > and I/O flow through it. So our intention of
> proposing
> > > the
> > > >> > > > "Storage
> > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > snapshots
> > > >> > while
> > > >> > > > > still
> > > >> > > > > > > > > keeping the original definition of volume snpashots
> > (ie
> > > >> > upload
> > > >> > > to
> > > >> > > > > sec
> > > >> > > > > > > > > storage).
> > > >> > > > > > > > >
> > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > backups.
> > > >> > They
> > > >> > > > > don't
> > > >> > > > > > > work
> > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> disks. I
> > > am
> > > >> all
> > > >> > > in
> > > >> > > > > for
> > > >> > > > > > a
> > > >> > > > > > > > > better backup solution which handles these use cases
> > and
> > > >> > > utilizes
> > > >> > > > > the
> > > >> > > > > > > > > storage's advanced features.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > >> > > > > > paul.an...@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > In the beginning... there were CloudStack
> snapshots
> > > and
> > > >> > they
> > > >> > > > were
> > > >> > > > > > > > > actually
> > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > snapshots.
> > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > point-in-time
> > > >> > > > > hypervisor
> > > >> > > > > > > > > > snapshots) and we started referring to the
> original
> > > >> > snapshots
> > > >> > > > as
> > > >> > > > > > > volume
> > > >> > > > > > > > > > snapshots.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> people
> > > use
> > > >> > > volume
> > > >> > > > > > > > snapshots
> > > >> > > > > > > > > > as backups. However you can't in-place restore
> > volume
> > > >> > > snapshots
> > > >> > > > > and
> > > >> > > > > > > if
> > > >> > > > > > > > > you
> > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > snapshots
> > > >> must
> > > >> > be
> > > >> > > > > done
> > > >> > > > > > in
> > > >> > > > > > > > > > series, meaning that the state across all of the
> > > >> volumes is
> > > >> > > > > > unlikely
> > > >> > > > > > > to
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > consistent.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > > options
> > > >> > > which
> > > >> > > > > > users
> > > >> > > > > > > > > might
> > > >> > > > > > > > > > expect as well options as to where they might be
> > > >> stored. In
> > > >> > > my
> > > >> > > > > > ideal
> > > >> > > > > > > > > world
> > > >> > > > > > > > > > they would also be able to leverage back-end
> > hardware
> > > >> (such
> > > >> > > as
> > > >> > > > > > > > Solidfire,
> > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > Commvault
> > > >> etc
> > > >> > to
> > > >> > > > > > > accelerate
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > process.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > >> > > > > > > > > > Paul Angus
> > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> |
> > > >> > > > > m:
> > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > e: *paul.an...@shapeblue.com <javascript:;>
> > > >> > <javascript:;>
> > > >> > > |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > > >> > > > > <javascript:;>%20%7C%20t:%20@cloudyangus>
> > > >> > > > > > | w:
> > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> 4HS
> > UK
> > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> England
> > &
> > > >> > Wales.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > > Services India LLP is a company incorporated in
> > India
> > > >> and
> > > >> > is
> > > >> > > > > > operated
> > > >> > > > > > > > > under
> > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > >> Consultoria
> > > >> > > Ltda
> > > >> > > > > is
> > > >> > > > > > a
> > > >> > > > > > > > > > company incorporated in Brasil and is operated
> under
> > > >> > license
> > > >> > > > from
> > > >> > > > > > > Shape
> > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > registered
> > > >> by
> > > >> > The
> > > >> > > > > > > Republic
> > > >> > > > > > > > of
> > > >> > > > > > > > > > South Africa and is traded under license from
> Shape
> > > Blue
> > > >> > Ltd.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > This email and any attachments to it may be
> > > confidential
> > > >> > and
> > > >> > > > are
> > > >> > > > > > > > intended
> > > >> > > > > > > > > > solely for the use of the individual to whom it is
> > > >> > addressed.
> > > >> > > > Any
> > > >> > > > > > > views
> > > >> > > > > > > > > or
> > > >> > > > > > > > > > opinions expressed are solely those of the author
> > and
> > > do
> > > >> > not
> > > >> > > > > > > > necessarily
> > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > companies.
> > > >> If
> > > >> > > you
> > > >> > > > > are
> > > >> > > > > > > not
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > intended recipient of this email, you must neither
> > > take
> > > >> any
> > > >> > > > > action
> > > >> > > > > > > > based
> > > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > Please
> > > >> > > > contact
> > > >> > > > > > the
> > > >> > > > > > > > > sender
> > > >> > > > > > > > > > if you believe you have received this email in
> > error.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mush...@gmail.com
> > > >> > > > > <javascript:;> <javascript:;>]
> > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > <javascript:;>
> > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Paul,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > When you say actual backups, how would it be
> > different
> > > >> from
> > > >> > > the
> > > >> > > > > > > Volume
> > > >> > > > > > > > > > Snapshots that exist currently. My understanding
> is
> > > that
> > > >> > > > Backups
> > > >> > > > > > end
> > > >> > > > > > > up
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > point-in-time
> > > >> > state
> > > >> > > of
> > > >> > > > > > your
> > > >> > > > > > > > > volume
> > > >> > > > > > > > > > which can be restored back correct?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -Syed
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > >> > > > > > > paul.an...@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Syed,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> > export
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > to
> > > >> > > > > > > > > > > secondary storage if the user requests a
> template
> > > from
> > > >> > the
> > > >> > > > > > snapshot
> > > >> > > > > > > > or
> > > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> > This
> > > >> is a
> > > >> > > > good,
> > > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > > storage
> > > >> is
> > > >> > > > super
> > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take
> up
> > > >> next
> > > >> > to
> > > >> > > no
> > > >> > > > > > > space.
> > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > general
> > > >> > > purpose
> > > >> > > > > API,
> > > >> > > > > > > and
> > > >> > > > > > > > > > > other storage systems may not be as awesome as
> > > Mike's.
> > > >> > > That's
> > > >> > > > > my
> > > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB
> to
> > > >> move
> > > >> > > from
> > > >> > > > > > > primary
> > > >> > > > > > > > > > > to sec storage and then create a VM template out
> > of
> > > it
> > > >> > may
> > > >> > > be
> > > >> > > > > too
> > > >> > > > > > > > long
> > > >> > > > > > > > > > for users.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > snapshot
> > > >> for
> > > >> > > > > backup,
> > > >> > > > > > > > it's
> > > >> > > > > > > > > > > just that users want to do backups and a volume
> > > >> snapshot
> > > >> > is
> > > >> > > > the
> > > >> > > > > > > only
> > > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> > and
> > > >> can
> > > >> > be
> > > >> > > > > > > scheduled.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > actual
> > > >> > backups
> > > >> > > > > > > (through
> > > >> > > > > > > > > > > recognised backup providers) and the user
> > > requirements
> > > >> > > around
> > > >> > > > > > them
> > > >> > > > > > > > > > > (particularly restoration use cases) as a
> separate
> > > >> thread
> > > >> > > of
> > > >> > > > > > work.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul
> > > >> Angus
> > > >> > > VP
> > > >> > > > > > > > Technology
> > > >> > > > > > > > > > > , ShapeBlue
> > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > |
> > > >> > > > > m:
> > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > e: *paul.an...@shapeblue.com <javascript:;>
> > > >> > > <javascript:;> |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > > >> <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> 4HS
> > > UK
> > > >> > Shape
> > > >> > > > > Blue
> > > >> > > > > > > Ltd
> > > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > > >> ShapeBlue
> > > >> > > > > Services
> > > >> > > > > > > > India
> > > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > > operated
> > > >> > > under
> > > >> > > > > > > license
> > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > >> Ltda
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > license
> > > >> from
> > > >> > > > Shape
> > > >> > > > > > > Blue
> > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > by
> > > >> The
> > > >> > > > > Republic
> > > >> > > > > > > of
> > > >> > > > > > > > > > > South Africa and is traded under license from
> > Shape
> > > >> Blue
> > > >> > > Ltd.
> > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > >> > > > > > > > > > > This email and any attachments to it may be
> > > >> confidential
> > > >> > > and
> > > >> > > > > are
> > > >> > > > > > > > > > > intended solely for the use of the individual to
> > > whom
> > > >> it
> > > >> > is
> > > >> > > > > > > > addressed.
> > > >> > > > > > > > > > > Any views or opinions expressed are solely those
> > of
> > > >> the
> > > >> > > > author
> > > >> > > > > > and
> > > >> > > > > > > do
> > > >> > > > > > > > > > > not necessarily represent those of Shape Blue
> Ltd
> > or
> > > >> > > related
> > > >> > > > > > > > > > > companies. If you are not the intended recipient
> > of
> > > >> this
> > > >> > > > email,
> > > >> > > > > > you
> > > >> > > > > > > > > > > must neither take any action based upon its
> > > contents,
> > > >> nor
> > > >> > > > copy
> > > >> > > > > or
> > > >> > > > > > > > show
> > > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > > believe
> > > >> > you
> > > >> > > > have
> > > >> > > > > > > > > received
> > > >> > > > > > > > > > this email in error.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > syed1.mush...@gmail.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > > <javascript:;>
> > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > Snapshots
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > > where
> > > >> > they
> > > >> > > > do
> > > >> > > > > > EBS
> > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > > >> followed
> > > >> > > > that.
> > > >> > > > > > And
> > > >> > > > > > > > as
> > > >> > > > > > > > > > > an end user who is oblivious to the internals of
> > my
> > > >> > > provider,
> > > >> > > > > my
> > > >> > > > > > > > > > > expectation would be something similar to what
> AWS
> > > as
> > > >> > that
> > > >> > > is
> > > >> > > > > my
> > > >> > > > > > > > > > > biggest reference point.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> Storage
> > > >> > failure
> > > >> > > on
> > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> motivations
> > > >> for us
> > > >> > > to
> > > >> > > > > push
> > > >> > > > > > > > data
> > > >> > > > > > > > > > > to secondary storage. Primary storage (atleast
> for
> > > us)
> > > >> > > costs
> > > >> > > > > > > around 3
> > > >> > > > > > > > > > > times as much as secondary storage and snapshots
> > on
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > customers
> > > >> who
> > > >> > > do
> > > >> > > > > > daily
> > > >> > > > > > > > > > backups).
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski
> <
> > > >> > > > > > > > > > > mike.tutkow...@solidfire.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > For most systems I've worked on, a snapshot
> and
> > a
> > > >> > backup
> > > >> > > > are
> > > >> > > > > > two
> > > >> > > > > > > > > > > > completely different things (but CloudStack
> has
> > > >> > > > traditionally
> > > >> > > > > > > used
> > > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > > >> though,
> > > >> > > that
> > > >> > > > > if
> > > >> > > > > > > your
> > > >> > > > > > > > > > > > primary storage is running on SolidFire that
> it
> > is
> > > >> > > unlikely
> > > >> > > > > > > you'll
> > > >> > > > > > > > > > > > encounter an issue where your primary storage
> > goes
> > > >> > > offline
> > > >> > > > > (and
> > > >> > > > > > > > > > > > you'll even maintain your performance
> guarantees
> > > >> during
> > > >> > > > > failure
> > > >> > > > > > > > > > > > scenarios and upgrades, as well). That being
> the
> > > >> case,
> > > >> > it
> > > >> > > > is
> > > >> > > > > > less
> > > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > > >> perfectly
> > > >> > > OK
> > > >> > > > if
> > > >> > > > > > > > that's
> > > >> > > > > > > > > > > > what we want to do
> > > >> > > > > > > > > > > here).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > >> > > > > > > > > > > > <syed1.mush...@gmail.com <javascript:;>
> > > >> > <javascript:;>>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Paul,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I believe with the current implementation of
> > > >> > Snapshots
> > > >> > > on
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> > to
> > > >> the
> > > >> > > > > secondary
> > > >> > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > While this solves the problem of having
> > > snapshots
> > > >> > > taking
> > > >> > > > > > > forever
> > > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > > >> > > > > > > > > > > > huge
> > > >> > > > > > > > > > > > > liability if our primary storage goes down.
> We
> > > see
> > > >> > > > > snapshots
> > > >> > > > > > as
> > > >> > > > > > > > > > > > > our recovery path because we store them in
> > Swift
> > > >> > which
> > > >> > > is
> > > >> > > > > > > > reliable
> > > >> > > > > > > > > > > > > and resilient to failures.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > > Volume
> > > >> > > > snapshots
> > > >> > > > > > > > always
> > > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > > >> Snapshots
> > > >> > > stay
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > primary
> > > >> > > > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > A provider could potentially mix both these
> > and
> > > >> solve
> > > >> > > the
> > > >> > > > > > > problem
> > > >> > > > > > > > > > > > > that you mentioned where you want to meet
> > user's
> > > >> > > > > expectation
> > > >> > > > > > > of a
> > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> > having
> > > >> an
> > > >> > > > ability
> > > >> > > > > > to
> > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> device)
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > -Syed
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > >> > > > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > HI guys,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of
> FS
> > > for
> > > >> > the
> > > >> > > > > > > > > > > > > > SAN-snapshot
> > > >> > > > > > > > > > > > feature
> > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > From my discussions with users and
> operators
> > > >> around
> > > >> > > > > > snapshots
> > > >> > > > > > > > > > > > > > I'd make
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > following observations:
> > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > > >> long-term
> > > >> > > and
> > > >> > > > > > short
> > > >> > > > > > > > > > > > > > term)
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > the expectation that they can use them for
> > > >> recovery
> > > >> > > if
> > > >> > > > > > > > required.
> > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > something
> > > >> > has
> > > >> > > > gone
> > > >> > > > > > > wrong
> > > >> > > > > > > > > > > > > > with primary storage.
> > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> export
> > > >> > > snapshots
> > > >> > > > as
> > > >> > > > > > > well
> > > >> > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > create
> > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> pain
> > > for
> > > >> > > > > operators,
> > > >> > > > > > I
> > > >> > > > > > > > > > > > > > know at
> > > >> > > > > > > > > > > > > least
> > > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > > take 2
> > > >> > days
> > > >> > > > to
> > > >> > > > > > > > > complete.
> > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used
> for
> > > >> > multiple
> > > >> > > > LVM
> > > >> > > > > > > > disks.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > > >> SolidFire
> > > >> > > > plugin
> > > >> > > > > > > (only
> > > >> > > > > > > > > > > > > > moving
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > > >> absolutely
> > > >> > > > have
> > > >> > > > > > to)
> > > >> > > > > > > is
> > > >> > > > > > > > > > > > > > a very
> > > >> > > > > > > > > > > > > good
> > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > problems
> > > >> an
> > > >> > > > > operator
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > > experience
> > > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > > >> storage
> > > >> > > pool
> > > >> > > > > in
> > > >> > > > > > a
> > > >> > > > > > > > > > cluster.
> > > >> > > > > > > > > > > > (I
> > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > > >> SolidFire
> > > >> > > case
> > > >> > > > > > Mike
> > > >> > > > > > > :)
> > > >> > > > > > > > > > > > > > ) And
> > > >> > > > > > > > > > > > if
> > > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > > slow,
> > > >> the
> > > >> > > > time
> > > >> > > > > to
> > > >> > > > > > > > > > > > > > being able
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > create a template or export the volume
> will
> > be
> > > >> > slow.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > So for me the issue is around making sure
> > that
> > > >> the
> > > >> > > end
> > > >> > > > > > users
> > > >> > > > > > > > > > > > expectations
> > > >> > > > > > > > > > > > > > are met (while improving the
> > speed/efficiency
> > > of
> > > >> > the
> > > >> > > > back
> > > >> > > > > > > end)
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > >> Paul
> > > >> > > > Angus
> > > >> > > > > > VP
> > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > >> > > > > > > > > > > > > >
> > > >> > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > > > > |
> > > >> > > > > > > > m:
> > > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > > > > e: *paul.an...@shapeblue.com
> <javascript:;>
> > > >> > > > > <javascript:;> | t:
> > > >> > > > > > @cloudyangus*
> > > >> > > > > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > > >> > > > <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > >> 4HS
> > > >> > UK
> > > >> > > > > Shape
> > > >> > > > > > > > Blue
> > > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > > >> Wales.
> > > >> > > > > ShapeBlue
> > > >> > > > > > > > > > > > > > Services India LLP is a company
> incorporated
> > > in
> > > >> > India
> > > >> > > > and
> > > >> > > > > > is
> > > >> > > > > > > > > > > > > > operated
> > > >> > > > > > > > > > > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > Brasil
> > > >> > > > > Consultoria
> > > >> > > > > > > Ltda
> > > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > > >> operated
> > > >> > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> > Pty
> > > >> Ltd
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > > > > registered by The Republic
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > South Africa and is traded under license
> > from
> > > >> Shape
> > > >> > > > Blue
> > > >> > > > > > Ltd.
> > > >> > > > > > > > > > > > > > ShapeBlue
> > > >> > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > > > > > This email and any attachments to it may
> be
> > > >> > > > confidential
> > > >> > > > > > and
> > > >> > > > > > > > are
> > > >> > > > > > > > > > > > intended
> > > >> > > > > > > > > > > > > > solely for the use of the individual to
> whom
> > > it
> > > >> is
> > > >> > > > > > addressed.
> > > >> > > > > > > > > > > > > > Any views
> > > >> > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > > >> author
> > > >> > and
> > > >> > > > do
> > > >> > > > > > not
> > > >> > > > > > > > > > > > necessarily
> > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> related
> > > >> > > companies.
> > > >> > > > > If
> > > >> > > > > > > you
> > > >> > > > > > > > > > > > > > are not
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > > >> neither
> > > >> > > take
> > > >> > > > > any
> > > >> > > > > > > > > > > > > > action
> > > >> > > > > > > > > > > > based
> > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > > >> anyone.
> > > >> > > > Please
> > > >> > > > > > > > contact
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > sender
> > > >> > > > > > > > > > > > > > if you believe you have received this
> email
> > in
> > > >> > error.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > >> pd...@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > >> > Snapshots
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi Mike,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > >> StorageSnapshot
> > > >> > > for
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > expected,
> > > >> > > > behavior
> > > >> > > > > > is
> > > >> > > > > > > to
> > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > Storage.
> > > >> So a
> > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> storage
> > > >> > > appliance.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > > see a
> > > >> > > strong
> > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> moved
> > > >> into
> > > >> > > > > secondary
> > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> > not
> > > >> > useful,
> > > >> > > > but
> > > >> > > > > > > both
> > > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> > remain
> > > >> on
> > > >> > the
> > > >> > > > > > managed
> > > >> > > > > > > > > > > > > > storage, and another type of
> > > >> > > > > > > > > > > > snapshot
> > > >> > > > > > > > > > > > > > that end up into the secondary storage.
> > Since
> > > >> > you've
> > > >> > > > > done a
> > > >> > > > > > > lot
> > > >> > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > work,
> > > >> > > > > > > > > > > > > > might easier to just add a parameter to
> the
> > > >> current
> > > >> > > > > > snapshot
> > > >> > > > > > > > API
> > > >> > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > snapshot
> > > >> into
> > > >> > > the
> > > >> > > > > > > > secondary
> > > >> > > > > > > > > > > > storage?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > PL
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > Tutkowski <
> > > >> > > > > > > > > > > > > > mike.tutkow...@solidfire.com
> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > I think that all sounds reasonable then
> -
> > > >> thanks!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > Mushtaq <
> > > >> > > > > > > > > > > > syed1.mush...@gmail.com <javascript:;>
> > > >> <javascript:;>>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > > >> > requirements.
> > > >> > > > One
> > > >> > > > > > of
> > > >> > > > > > > > our
> > > >> > > > > > > > > > > > earlier
> > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > argument
> > > to
> > > >> > the
> > > >> > > > > create
> > > >> > > > > > > > > > > > > > >> snapshot
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > volume
> > > to
> > > >> > sec
> > > >> > > > > > storage
> > > >> > > > > > > > but
> > > >> > > > > > > > > > > > > > >> we realized it would make management of
> > > >> > snapshots
> > > >> > > > > quite
> > > >> > > > > > > > messy
> > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > Tutkowski
> > > >> > > > > > > > > > > > > > >> <mike.tutkow...@solidfire.com
> > > <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >> wrote:
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >>> Hi,
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > > >> > > requirements
> > > >> > > > > > here:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> storage
> > > (1:1
> > > >> > > > mapping
> > > >> > > > > > > > between
> > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > volume).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > (introduced
> > > >> in
> > > >> > > 4.6)
> > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> snapshot
> > on
> > > >> the
> > > >> > > SAN,
> > > >> > > > > and
> > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > > >> snapshot,
> > > >> > but
> > > >> > > > to
> > > >> > > > > > copy
> > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > > NFS.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> > It's
> > > >> the
> > > >> > > > backup
> > > >> > > > > to
> > > >> > > > > > > NFS
> > > >> > > > > > > > > > > > > > >>> that is optional.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > > >> snapshot
> > > >> > > > that's
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> > from
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > > > > > > > > >>> that's on
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > > >> > CloudStack
> > > >> > > > > volume
> > > >> > > > > > > > ends
> > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > snapshot
> > > >> API
> > > >> > > > then
> > > >> > > > > or
> > > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > > >> snapshot
> > > >> > > > > without
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> backup to
> > > >> > > > > > > > > > > NFS?
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Thanks!
> > > >> > > > > > > > > > > > > > >>> Mike
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > > Mushtaq
> > > >> > > > > > > > > > > > > > >>> <syed1.mush...@gmail.com
> <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >>> wrote:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > functionalities
> > > >> > > > > (snapshot
> > > >> > > > > > on
> > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > > would
> > > >> > like
> > > >> > > > to
> > > >> > > > > > have
> > > >> > > > > > > > > both.
> > > >> > > > > > > > > > > > > > >>>> For example, some of our customers
> want
> > > to
> > > >> > > > implement
> > > >> > > > > > > their
> > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> encryption
> > > to
> > > >> > their
> > > >> > > > > > backups
> > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> Storage
> > > >> > Snapshot
> > > >> > > > > while
> > > >> > > > > > > our
> > > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> > the
> > > >> > > standard
> > > >> > > > > > > volume
> > > >> > > > > > > > > > > snapshot.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> To keep things backward compatible,
> we
> > > can
> > > >> > add a
> > > >> > > > > > setting
> > > >> > > > > > > > > > > > > > >>>> which
> > > >> > > > > > > > > > > > says
> > > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > > >> because,
> > > >> > > after
> > > >> > > > > > all,
> > > >> > > > > > > > you
> > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > > doing
> > > >> a
> > > >> > > > Volume
> > > >> > > > > > > > > Snapshot.
> > > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> > not
> > > do
> > > >> > the
> > > >> > > > > > upload.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> What do you think about this
> approach?
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > >> > > > > > > > > > > > > > >>>> -Syed
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > > >> > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>> mike.tutkow...@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> load
> > > >> here,
> > > >> > but
> > > >> > > > if
> > > >> > > > > a
> > > >> > > > > > > > given
> > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually
> need
> > > to
> > > >> > > provide
> > > >> > > > > > both
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > ability
> > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> > to
> > > >> NFS
> > > >> > (if
> > > >> > > > > just
> > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then
> we
> > > >> might
> > > >> > be
> > > >> > > > able
> > > >> > > > > > to
> > > >> > > > > > > > get
> > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > custom
> > > >> > > snapshot
> > > >> > > > > > > strategy
> > > >> > > > > > > > > > > > > > >>>>> and
> > > >> > > > > > > > > > > > data
> > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> > where
> > > >> the
> > > >> > > > > > CloudStack
> > > >> > > > > > > > > > > > > > >>>>> cloud
> > > >> > > > > > > > > > > > does
> > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > >> exported-to-NFS
> > > >> > > > > backup.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> behavior
> > > >> would
> > > >> > be
> > > >> > > > to
> > > >> > > > > > use
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > > strategy
> > > >> > that
> > > >> > > > we
> > > >> > > > > > > > already
> > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > > >> > snapshot).
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> > wants
> > > >> to
> > > >> > > take
> > > >> > > > a
> > > >> > > > > > SAN
> > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported
> to
> > > >> NFS,
> > > >> > > then
> > > >> > > > we
> > > >> > > > > > > could
> > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> > file
> > > to
> > > >> > make
> > > >> > > > use
> > > >> > > > > > of
> > > >> > > > > > > a
> > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> motion
> > > >> > strategy
> > > >> > > > that
> > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > >> > > > > > > > > > > > > > activities.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> > the
> > > >> > default
> > > >> > > > for
> > > >> > > > > > > > users,
> > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> this
> > > >> > behavior
> > > >> > > > via
> > > >> > > > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> Mike
> > > >> > > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>>> mike.tutkow...@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> come
> > up
> > > >> > with a
> > > >> > > > > > viable
> > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> > for
> > > >> them
> > > >> > > to
> > > >> > > > > move
> > > >> > > > > > > > from
> > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> some
> > > >> obvious
> > > >> > > way
> > > >> > > > > > that
> > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > do this).
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM,
> Syed
> > > >> > Mushtaq
> > > >> > > <
> > > >> > > > > > > > > > > > > > >>>>>> syed1.mush...@gmail.com
> > <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > upgrade
> > > >> path
> > > >> > > > > however,
> > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and
> are
> > > on a
> > > >> > > > managed
> > > >> > > > > > > > storage
> > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > >> functionality
> > > >> > > > with
> > > >> > > > > > > Volume
> > > >> > > > > > > > >
>
>
>
> --
> Ian Rae
> CEO | PDG
> c: 514.944.4008
>
> CloudOps | Cloud Infrastructure and Networking Solutions
> www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
