Do you have any logging running on those OSDs? I'm going to need to get somebody else to look at this, but if we could check the probe messages being sent that might be helpful. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, Apr 15, 2014 at 4:36 PM, Craig Lewis <cle...@centraldesktop.com> wrote: > http://pastebin.com/ti1VYqfr > > I assume the problem is at the very end: > "probing_osds": [ > 0, > 2, > 3, > 4, > 11, > 13], > "down_osds_we_would_probe": [], > "peering_blocked_by": []}, > > > OSDs 3, 4, and 11 have been UP and IN for hours. OSDs 0, 2, and 13 have > been UP and IN since the problems started, but they never complete probing. > > > > > Craig Lewis > Senior Systems Engineer > Office +1.714.602.1309 > Email cle...@centraldesktop.com > > Central Desktop. Work together in ways you never thought possible. > Connect with us Website | Twitter | Facebook | LinkedIn | Blog > > On 4/15/14 16:07 , Gregory Farnum wrote: > > What are the results of "ceph osd pg 11.483 query"? > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Tue, Apr 15, 2014 at 4:01 PM, Craig Lewis <cle...@centraldesktop.com> > wrote: > > I have 1 incomplete PG. The data is gone, but I can upload it again. I > just need to make the cluster start working so I can upload it. > > I've read a bunch of mailling list posts, and found ceph pg force_create_pg. > Except, it doesn't work. > > I run: > root@ceph1c:/var/lib/ceph/osd# ceph pg force_create_pg 11.483 > pg 11.483 now creating, ok > > The incomplete PG switches to creating. It sits in creating for a while, > then flips back to incomplete: > 2014-04-15 15:06:11.876535 mon.0 [INF] pgmap v5719605: 2592 pgs: 2586 > active+clean, 1 incomplete, 5 active+clean+scrubbing+deep; 15086 GB data, > 27736 GB used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:13.899681 mon.0 [INF] pgmap v5719606: 2592 pgs: 1 creating, > 2586 active+clean, 5 active+clean+scrubbing+deep; 15086 GB data, 27736 GB > used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:14.965676 mon.0 [INF] pgmap v5719607: 2592 pgs: 1 creating, > 2586 active+clean, 5 active+clean+scrubbing+deep; 15086 GB data, 27736 GB > used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:15.995570 mon.0 [INF] pgmap v5719608: 2592 pgs: 1 creating, > 2586 active+clean, 5 active+clean+scrubbing+deep; 15086 GB data, 27736 GB > used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:17.019972 mon.0 [INF] pgmap v5719609: 2592 pgs: 1 creating, > 2586 active+clean, 5 active+clean+scrubbing+deep; 15086 GB data, 27736 GB > used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:18.048487 mon.0 [INF] pgmap v5719610: 2592 pgs: 1 creating, > 2586 active+clean, 5 active+clean+scrubbing+deep; 15086 GB data, 27736 GB > used, 28127 GB / 55864 GB avail > 2014-04-15 15:06:19.093757 mon.0 [INF] pgmap v5719611: 2592 pgs: 2586 > active+clean, 1 incomplete, 5 active+clean+scrubbing+deep; 15086 GB data, > 27736 GB used, 28127 GB / 55864 GB avail > > I'm on: > root@ceph0c:~# ceph -v > ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) > > root@ceph0c:~# uname -a > Linux ceph0c 3.5.0-46-generic #70~precise1-Ubuntu SMP Thu Jan 9 23:55:12 UTC > 2014 x86_64 x86_64 x86_64 GNU/Linux > > root@ceph0c:~# cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=12.04 > DISTRIB_CODENAME=precise > DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS" > > -- > > Craig Lewis > Senior Systems Engineer > Office +1.714.602.1309 > Email cle...@centraldesktop.com > > Central Desktop. Work together in ways you never thought possible. > Connect with us Website | Twitter | Facebook | LinkedIn | Blog > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
