I don't see anything related to lost objects in your output. I just see waiting on backfill, backfill_toofull, remapped, and so forth. You can read a bit about what is going on here: http://ceph.com/docs/next/rados/operations/monitoring-osd-pg/
Keep us posted as to the recovery, and let me know what I can do to improve the docs for scenarios like this. On Sat, Apr 20, 2013 at 10:52 AM, Marco Aroldi <marco.aro...@gmail.com>wrote: > John, > thanks for the quick reply. > Below you can see my ceph osd tree > The problem is caused not by the failure itself, but by the "renamed" > bunch of devices. > It was like a deadly 15-puzzle > I think that the solution was to mount the devices in fstab using UUID > (/dev/disk/by-uuid) instead of /dev/sdX > > However, yes I have an entry in my ceph.conf (devs = /dev/sdX1 -- > osd_journal = /dev/sdX2) *and* an entry in my fstab for each OSD > > The node with failed disk is s103 (osd.59) > > Now i have 5 osd from s203 up and in to try to let ceph rebalance > data... but is still a bloody mess. > Look at ceph -w output: is reported a total of 110TB: is wrong... al > drives are 2TB and i have 49 drives up and in -- total 98Tb > I think that 110TB (55 osd) was the size before cluster became inaccessible > > # id weight type name up/down reweight > -1 130 root default > -9 65 room p1 > -3 44 rack r14 > -4 22 host s101 > 11 2 osd.11 up 1 > 12 2 osd.12 up 1 > 13 2 osd.13 up 1 > 14 2 osd.14 up 1 > 15 2 osd.15 up 1 > 16 2 osd.16 up 1 > 17 2 osd.17 up 1 > 18 2 osd.18 up 1 > 19 2 osd.19 up 1 > 20 2 osd.20 up 1 > 21 2 osd.21 up 1 > -6 22 host s102 > 33 2 osd.33 up 1 > 34 2 osd.34 up 1 > 35 2 osd.35 up 1 > 36 2 osd.36 up 1 > 37 2 osd.37 up 1 > 38 2 osd.38 up 1 > 39 2 osd.39 up 1 > 40 2 osd.40 up 1 > 41 2 osd.41 up 1 > 42 2 osd.42 up 1 > 43 2 osd.43 up 1 > -13 21 rack r10 > -12 21 host s103 > 55 2 osd.55 up 0 > 56 2 osd.56 up 0 > 57 2 osd.57 up 0 > 58 2 osd.58 up 0 > 59 2 osd.59 down 0 > 60 2 osd.60 down 0 > 61 2 osd.61 down 0 > 62 2 osd.62 up 0 > 63 2 osd.63 up 0 > 64 1.5 osd.64 up 0 > 65 1.5 osd.65 down 0 > -10 65 room p2 > -7 22 rack r20 > -5 22 host s202 > 22 2 osd.22 up 1 > 23 2 osd.23 up 1 > 24 2 osd.24 up 1 > 25 2 osd.25 up 1 > 26 2 osd.26 up 1 > 27 2 osd.27 up 1 > 28 2 osd.28 up 1 > 29 2 osd.29 up 1 > 30 2 osd.30 up 1 > 31 2 osd.31 up 1 > 32 2 osd.32 up 1 > -8 22 rack r22 > -2 22 host s201 > 0 2 osd.0 up 1 > 1 2 osd.1 up 1 > 2 2 osd.2 up 1 > 3 2 osd.3 up 1 > 4 2 osd.4 up 1 > 5 2 osd.5 up 1 > 6 2 osd.6 up 1 > 7 2 osd.7 up 1 > 8 2 osd.8 up 1 > 9 2 osd.9 up 1 > 10 2 osd.10 up 1 > -14 21 rack r21 > -11 21 host s203 > 44 2 osd.44 up 1 > 45 2 osd.45 up 1 > 46 2 osd.46 up 1 > 47 2 osd.47 up 1 > 48 2 osd.48 up 1 > 49 2 osd.49 up 0 > 50 2 osd.50 up 0 > 51 2 osd.51 up 0 > 52 1.5 osd.52 up 0 > 53 1.5 osd.53 up 0 > 54 2 osd.54 up 0 > > > ceph -w > > 2013-04-20 19:46:48.608988 mon.0 [INF] pgmap v1352767: 17280 pgs: 58 > active, 12581 active+clean, 1686 active+remapped+wait_backfill, 24 > active+degraded+wait_backfill, 224 > active+remapped+wait_backfill+backfill_toofull, 1061 > active+recovery_wait, 4 > active+degraded+wait_backfill+backfill_toofull, 629 peering, 626 > active+remapped, 72 active+remapped+backfilling, 89 active+degraded, > 14 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 8 > active+degraded+remapped+wait_backfill, 20 > active+recovery_wait+remapped, 5 > active+degraded+remapped+wait_backfill+backfill_toofull, 162 > remapped+peering, 1 active+degraded+remapped+backfilling, 2 > active+degraded+remapped+backfill_toofull, 13 active+recovering; 49777 > GB data, 72863 GB used, 40568 GB / 110 TB avail; 2965687/21848501 > degraded (13.574%); recovering 5 o/s, 16363B/s > > 2013/4/20 John Wilkins <john.wilk...@inktank.com>: > > Marco, > > > > If you do a "ceph tree" can you see if your OSDs are all up? You seem to > > have at least one problem related to the backfill OSDs being too full, > and > > some which are near full or full for the purposes of storage. See the > > following in the documentation to see if this helps: > > > > > http://ceph.com/docs/master/rados/configuration/mon-config-ref/#storage-capacity > > > http://ceph.com/docs/master/rados/configuration/osd-config-ref/#backfilling > > > http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#no-free-drive-space > > > > Before you start deleting data as a remedy, you'd want to at least try to > > get the OSDs back up and running first. > > > > If rebooting changed the drive names, you might look here: > > > http://ceph.com/docs/master/rados/configuration/osd-config-ref/#general-settings > > > > We have default settings for OSD and journal paths, which you could > override > > if you can locate the data and journal sources on the renamed drives. If > you > > mounted them, but didn't add them to the fstab, that might be the source > of > > the problem. I'd rather see you use the default paths, as it would be > easier > > to troubleshoot later. So did you mount the drives, but not add the mount > > points to fstab? > > > > John > > > > > > > > > > On Sat, Apr 20, 2013 at 8:46 AM, Marco Aroldi <marco.aro...@gmail.com> > > wrote: > >> > >> Hi, > >> due a harware failure during expanding ceph, I'm in big trouble > >> because the cephfs doesn't mount anymore. > >> I was adding a couple storage nodes, but a disk has failed and after a > >> reboot the OS (ubuntu 12.04) renamed the remaining devices, so the > >> entire node has been screwed out. > >> > >> Now, from the "sane new node", I'm taking some new osd up and in > >> because the cluster is near full and I can't revert completely the > >> situation as before > >> > >> *I can* afford data loss, but i need to regain access to the filesystem > >> > >> My setup: > >> 3 mon + 3 mds > >> 4 storage nodes (i was adding no. 5 and 6) > >> > >> Ceph 0.56.4 > >> > >> > >> ceph health: > >> HEALTH_ERR 2008 pgs backfill; 246 pgs backfill_toofull; 74 pgs > >> backfilling; 134 pgs degraded; 790 pgs peering; 10 pgs recovering; > >> 1116 pgs recovery_wait; 790 pgs stuck inactive; 4782 pgs stuck > >> unclean; recovery 3049459/21926624 degraded (13.908%); recovering 6 > >> o/s, 16316KB/s; 4 full osd(s); 30 near full osd(s); full,noup,nodown > >> flag(s) set > >> > >> > >> > >> ceph mds dump: > >> dumped mdsmap epoch 44 > >> epoch 44 > >> flags 0 > >> created 2013-03-18 14:42:29.330548 > >> modified 2013-04-20 17:14:32.969332 > >> tableserver 0 > >> root 0 > >> session_timeout 60 > >> session_autoclose 300 > >> last_failure 43 > >> last_failure_osd_epoch 18160 > >> compat compat={},rocompat={},incompat={1=base v0.20,2=client > >> writeable ranges,3=default file layouts on dirs,4=dir inode in > >> separate object} > >> max_mds 1 > >> in 0 > >> up {0=6376} > >> failed > >> stopped > >> data_pools [0] > >> metadata_pool 1 > >> 6376: 192.168.21.11:6800/13457 'm1' mds.0.9 up:replay seq 1 > >> 5945: 192.168.21.13:6800/12999 'm3' mds.-1.0 up:standby seq 1 > >> 5963: 192.168.21.12:6800/22454 'm2' mds.-1.0 up:standby seq 1 > >> > >> > >> > >> ceph mon dump: > >> epoch 1 > >> fsid d634f7b3-8a8a-4893-bdfb-a95ccca7fddd > >> last_changed 2013-03-18 14:39:42.253923 > >> created 2013-03-18 14:39:42.253923 > >> 0: 192.168.21.11:6789/0 mon.m1 > >> 1: 192.168.21.12:6789/0 mon.m2 > >> 2: 192.168.21.13:6789/0 mon.m3 > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > -- > > John Wilkins > > Senior Technical Writer > > Intank > > john.wilk...@inktank.com > > (415) 425-9599 > > http://inktank.com > -- John Wilkins Senior Technical Writer Intank john.wilk...@inktank.com (415) 425-9599 http://inktank.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
