On 7/24/19 6:02 PM, Sinan Polat wrote:
> Hi,
>
> Why not using backup tools that can do native OpenStack backups?
>
> We are also using Ceph as the cinder backend on our OpenStack platform. We
> use CommVault to make our backups.
How much data is there in that Ceph cluster? And how does it perform?
I assume it exports the RBD images through Cinder and stores them
somewhere? But this is probably a complete copy of the RBD image and I
just wonder how that scales.
A lot of systems are >500TB and it's very difficult to back that up
every week or so.
Wido
>
> - Sinan
>
>> Op 24 jul. 2019 om 17:48 heeft Wido den Hollander <w...@42on.com> het
>> volgende geschreven:
>>
>>
>>
>>> On 7/24/19 4:06 PM, Fabian Niepelt wrote:
>>> Hi, thanks for the reply.
>>>
>>> Am Mittwoch, den 24.07.2019, 15:26 +0200 schrieb Wido den Hollander:
>>>>
>>>> On 7/24/19 1:37 PM, Fabian Niepelt wrote:
>>>>> Hello ceph-users,
>>>>>
>>>>> I am currently building a Ceph cluster that will serve as a backend for
>>>>> Openstack and object storage using RGW. The cluster itself is finished and
>>>>> integrated with Openstack and virtual machines for testing are being
>>>>> deployed.
>>>>> Now I'm a bit stumped on how to effectively backup the Ceph pools.
>>>>> My requirements are two weekly backups, of which one must be offline after
>>>>> finishing backing up (systems turned powerless). We are expecting about
>>>>> 250TB to
>>>>> 500TB of data for now. The backups must protect against accidental pool
>>>>> deletion/corruption or widespread infection of a cryptovirus. In short:
>>>>> Complete
>>>>> data loss in the production Ceph cluster.
>>>>>
>>>>> At the moment, I am facing two issues:
>>>>>
>>>>> 1. For the cinder pool, I looked into creating snapshots using the ceph
>>>>> CLI
>>>>> (so
>>>>> they don't turn up in Openstack and cannot be accidentally deleted by
>>>>> users)
>>>>> and
>>>>> exporting their diffs. But volumes with snapshots created this way cannot
>>>>> be
>>>>> removed from Openstack. Does anyone have an idea how to do this better?
>>>>
>>>> You mean that while you leave the snapshot there OpenStack can't remove it?
>>>
>>> Yes, that is correct. cinder-volume cannot remove a volume that still has a
>>> snapshot. If the snapshot is created by openstack, it will remove the
>>> snapshot
>>> before removing the volume. But snapshotting directly from ceph will forego
>>> Openstack so it will never know about that snapshot's existence.
>>>
>>
>> Ah, yes. That means you would need to remove it manually.
>>
>>>>> Alternatively, I could do a full export each week, but I am not sure if
>>>>> that
>>>>> would be fast enough..
>>>>>
>>>>
>>>> It probably won't, but the full backup is still the safest way imho.
>>>> However: Does this scale?
>>>>
>>>> You can export multiple RBD images in parallel and store them somewhere
>>>> else, but it will still take a long time.
>>>>
>>>> The export needs to be stored somewhere and then picked up. Or you could
>>>> use some magic with Netcat to stream the RBD export to a destination host.
>>>>
>>>
>>> Scaling is also my biggest worry about this.
>>>
>>>>> 2. My search so far has only turned up backing up RBD pools, but how
>>>>> could I
>>>>> backup the pools that are used for object storage?
>>>>>
>>>>
>>>> Not easily. I think you mean RGW? You could try the RGW MultiSite, but
>>>> it's difficult.
>>>>
>>>> A complete DR with Ceph to restore it back to how it was at a given
>>>> point in time is a challenge.
>>>>
>>>
>>> Yes, I would like to backup the pools used by the RGW.
>>
>> Not really an option. You would need to use the RGW MultiSite to
>> replicate all data to a second environment.
>>
>>>
>>>>> Of course, I'm also open to completely other ideas on how to backup Ceph
>>>>> and
>>>>> would appreciate hearing how you people are doing your backups.
>>>>
>>>> A lot of time the backups are created inside the VMs on File level. And
>>>> there is a second OpenStack+Ceph system which runs a mirror of the VMs
>>>> or application. If one burns down it's not the end of the world.
>>>>
>>>> Trying to backup a Ceph cluster sounds very 'enterprise' and is
>>>> difficult to scale as well.
>>>>
>>>
>>> Are those backups saved in Ceph as well? I cannot solely rely on Ceph
>>> because we
>>> want to protect ourselves against failures in Ceph or a human accidentally
>>> or
>>> maliciously deletes all pools.
>>>
>>
>> That second OpenStack+Ceph environment is completely different. All the
>> VMs are set up twice and using replication and backups on application
>> level such things are redundant.
>>
>> Think about MySQL replication for example.
>>
>>> From what I'm reading, it seems to be better to maybe implement a backup
>>> solution outside of Ceph that our Openstack users can use and not deal with
>>> backing up Ceph at all, except its configs to get it running after total
>>> desaster...
>>>
>>
>> You could backup OpenStack's MySQL database, the ceph config and then
>> backup the data inside the VMs.
>>
>> It's very difficult to backup data for DR to a certain point of time
>> when you go into the >100TB scale.
>>
>> Wido
>>
>>>> Wido
>>>>
>>>>> Any help is much appreciated.
>>>>>
>>>>> Greetings
>>>>> Fabian
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
