You saved me a bunch of time; I was planning to test my backup and
restore later today. Thanks!
It occurred to me that the backups won't be as useful as I thought. I'd
need to make sure that the PGs hadn't moved around after the backup was
made. If they had, I'd spend a lot of time tracking down the new
locations and manually rsyncing data. Not a big deal on a small
cluster, but it get harder as the cluster gets larger. Now that I look
at the dumps, it looks like PG locations are one of the things missing.
Binary backups of the MON directories are fine. All of the problems
I've seen on the list occurred during cluster upgrades, so I'll make the
backup part of my upgrade procedure instead of a cron.
If I wanted to restore a backup, what would be required? Looking at
http://eu.ceph.com/docs/v0.47.1/ops/manage/grow/mon/#removing-a-monitor-from-an-unhealthy-or-down-cluster,
I don't see a monmap directory inside /var/lib/ceph/mon/ceph-#/
anymore. I assume it went away during the switch to LevelDB, so I think
I'll need to dump a copy when I make the binary backup.
I'm assuming there is some node specific data in each MON's store. If
not, could I just stop all monitors, rsync the backup to all monitors,
and start them all up?
I'm not using cephx, but I assume the keyrings would complicate things.
It would probably be easiest to make binary backups on all of the
monitors (with some time delay, so only one is offline at a time), then
start them up in newest-to-oldest-backup order. Or I could use LVM
snapshots to make simultaneous backups on all monitors.
Thanks for the info.
*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com <mailto:cle...@centraldesktop.com>
*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>
On 8/12/13 14:39 , Joao Eduardo Luis wrote:
On 08/08/13 15:21, Craig Lewis wrote:
I've seen a couple posts here about broken clusters that had to repair
by modifing the monmap, osdmap, or the crush rules.
The old school sysadmin in me says it would be a good idea to make
backups of these 3 databases. So far though, it seems like everybody
was able to repair their clusters by dumping the current map and
modifying it.
I'll probably do it, just to assuage my paranoia, but I was wondering
what you guys thought.
Well, this could get you *some* infos, but you wouldn't be able to
reconstruct a monitor this way. There's just way too many maps that
you'd need to reconstruct the monitor.
The not-so-best approach would be to grab all map epochs, from 1 to
the map's current epoch. We don't currently have a way to expose to
the user what is the first available map epoch in the store (the need
for it never came up), so for now you'd have to start at 1 and
increment it until you'd find an existing version (we trim old
versions, so that could be at 1, 10k, or a few hundred thousands,
depending on how many maps you have). With all that information, you
could somehow reconstruct a monitor with some effort -- and even so,
we currently only expose an interface to obtain maps for some services
such as the mon, osd, pg and mds; we have a bunch of other versions
kept in the monitor that are not currently exposed to the user.
This is something we definitely want to improve on, but as of this
moment the best approach to backup monitors reliably would be to stop
the monitor, copy the store, and restart the monitor. Assuming you
have 3+ monitors, stopping just one of them wouldn't affect the quorum
or cluster availability. And assuming you're backing up a monitor
that is in the quorum, then backing it up is as good as backing any
other monitor.
Hope this helps.
-Joao
I'm thinking of cronning this on the MON servers:
#!/usr/bin/env bash
# Number of days to keep backups
cleanup_age="10"
# Fetch the current timestamp, to use in the backup filenames
date=$(date +"%Y-%m-%dT%H:%M:%S")
# Dump the current maps
cd /var/lib/ceph/backups/
ceph mon getmap -o ./monmap.${date}
ceph osd getmap -o ./osdmap.${date}
ceph osd getcrushmap -o ./crushmap.${date}
# Delete old maps
find . -type f -regextype posix-extended -regex
'\./(mon|osd|crush)map\..*' -mtime +${cleanup_age} -print0 | xargs -0 rm
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
