Dear All, This is a great discussion. If I understand this correctly, this is a proposal for data protection as a whole for the OpenStack cloud, however this is not yet an official "incubation" request. We are having a good discussion on how we can better serve the adoption of OpenStack.
Having said that, the proposal will reuse the existing API and contributions by the community that are already in place. For example, Catlin's point is very valid... the Cinder storage vendor knows the best way to implement snapshots for their storage platforms. No doubt, Raksha should be leveraging that IP. Similarly Raksha will be leveraging Nova, Swift as well as Glance. Don't forget Neutron.... networking is very critical part of data protection for any VM or set of VMs. No one project has one single answer for a comprehensive data protection. The capabilities for backup and recovery exist in silos in various projects... 1. Images are backed-up by Nova 2. Volumes are backed-up by Cinder 3. I am not aware of a solution that can backup network configuration. 4. Not sure if we have something that can backup the resources of a VM ( vCPUs, Memory Configuration etc.) 5. One can't schedule and automate the above very easily. Ronen's point about consistency groups is right on the mark. We need to treat an application as an unit that may span multiple VMs, one or more images and one or more volumes. Just to reiterate, some form of these capabilities exist in the current projects, however as a user of OpenStack, I may not have a simple one click solution. With this proposal, the ask is to engage in a discussion to address the above use cases. IMHO we are on the right track with these discussions. We will be submitting the code in few days and looking forward for your feedback. I would also suggest a design summit where we can flush out more details. Regards, Giri -----Original Message----- From: Avishay Traeger [mailto:avis...@il.ibm.com] Sent: Saturday, August 31, 2013 10:53 PM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] Proposal for Raksha, a Data Protection As a Service project +1 From: Ronen Kat/Haifa/IBM@IBMIL To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>, Date: 09/01/2013 08:02 AM Subject: Re: [openstack-dev] Proposal for Raksha, a Data Protection As a Service project Hi Murali, Thanks for answering. I think the issues you raised indeed make sense, and important ones. We need to provide backup both for: 1. Volumes 2. VM instances (VM image, VM metadata, and attached volumes) While the Cinder-backup handles (1), and is a very mature service, it does not provide (2), and for Cinder-backup it does not make sense to handle (2) as well. Backup of VMs (as a package) is beyond the scope of Cinder, which implies that indeed something beyond Cinder should take this task. I think this can be done by having Nova orchestrate or assist the backup, either of volumes or VMs. I think that from a backup perspective, there is also a need for "consistency groups" - the set of entities (volumes) that are considered a single logical unit and should be backup together. This logical consistency group could be larger than a VM, but a VM is a good starting point. In any case, we should adopt the "off-load" approach: 1. Handle application consistency issues using Nova as it manages the VMs. Add to Nova functionality to support live and consistent backup - including orchestrating volume backup using Cinder 2. Have Cinder do the volume "backup", and Cinder then can delegate the task to the Storage/hypervisor or anyone else who provide a backup driver While a new project is a neat package that addresses the issues, but does it worth the work? OpenStack projects are complex, and successful projects require a lot of work and long-term maintenance, which is the real pain for open source projects, as the team tend to be very dynamic. My two cents is to adopt the "nova-networking" and "nova-volume" approach, try to extend the current work within Nova and Cinder, and if we find out it does not make sense anymore, explain the issues, and split the work to a new project. This way it, if a backup project is indeed needed, you already have the community to support the effort, and you already have a mature solution. Regards, __________________________________________ Ronen I. Kat Storage Research IBM Research - Haifa Phone: +972.3.7689493 Email: ronen...@il.ibm.com From: Caitlin Bestler <caitlin.best...@nexenta.com> To: Murali Balcha <murali.bal...@triliodata.com>, Cc: OpenStack Development Mailing List <openstack-dev@lists.openstack.org> Date: 31/08/2013 07:25 AM Subject: Re: [openstack-dev] Proposal for Raksha, a Data Protection As a Service project On 8/30/2013 12:49 PM, Murali Balcha wrote: > Hi Caitlin, > Did you get a chance to look at the wiki? It describes the raksha functionality in detail. > It includes more than volume backup. It includes vm images, all > volumes > and network configurations associated with vms and it > supports > incremental backups too. Volume backup is essential for > implementing > backup solution but not necessarily sufficient. Cinder already allows backing volumes up to Swift, and in fact allows incremental backups. Any code you write will not back up a vendor's volume more efficiently than the vendor's code itself can. The vendor's knowledge of how the data is stored is probably sufficient, but in this case a vendor has a far more powerful advantage. The vendor can transfer the volume directly to the Swift server. Your service, since it running on a compute node rather than the vendor's box, will first have to fetch the content and *then* send it to Swift. That's twice as much network traffic. This is not trivial when volumes are big, which they tend to be. If this service is implemented, customer who are using vendor backends such as NexentaStor, Netapp or CEPH will see their performance drop. That will clearly be unacceptable. New featqures are not allowed to trash existing performance, especially when they are not actually providing any new service to customers who already have volume backends with these features. You would need to have a proposal to work with the existing Cinder backend Volume Drivers that in no way removed any option vendors have currently to optimize performance. Doing that in a new project, rather than within Cinder, can only make life harder on the vendors and discourage participation in OpenStack. I believe all of the features you are looking at can be accomodated by taskflows using the existing Volume Driver feature (as evolving) in Cinder. A new project is not justified, and it will risk creating a major performance regression for some customers. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
