Re: [ceph-users] OSD issue: unable to obtain rotating service keys

2016-06-01 Thread Christian Balzer
Hello, On Wed, 1 Jun 2016 20:21:29 -0500 Jeffrey McDonald wrote: > Thanks Christian, > I did google the error and I actually found this link. (Of course, I > wouldn't want to waste others' time as either.) It appears to me to be a > different issue than what I see because the OSDs actually fa

Re: [ceph-users] OSD issue: unable to obtain rotating service keys

2016-06-01 Thread Jeffrey McDonald
Thanks Christian, I did google the error and I actually found this link. (Of course, I wouldn't want to waste others' time as either.) It appears to me to be a different issue than what I see because the OSDs actually fail to start. Anyways, after a few minutes I restarted the OSDs and they s

Re: [ceph-users] OSD issue: unable to obtain rotating service keys

2016-06-01 Thread Christian Balzer
Hello, On Wed, 1 Jun 2016 12:31:41 -0500 Jeffrey McDonald wrote: > Hi, > > I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from > ceph version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the > OSDs, I receive the error message: > Unfortunately (despite what common s

Re: [ceph-users] OSD issue: unable to obtain rotating service keys

2016-06-01 Thread Shinobu Kinjo
Would you enable debug for osd.177 debug osd = 20 debug filestore = 20 debug ms = 1 Cheers, Shinobu On Thu, Jun 2, 2016 at 2:31 AM, Jeffrey McDonald wrote: > Hi, > > I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from ceph > version to0.94.6-1trusty to 0.94.7-1trusty. Upo

[ceph-users] OSD issue: unable to obtain rotating service keys

2016-06-01 Thread Jeffrey McDonald
Hi, I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from ceph version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the OSDs, I receive the error message: 2016-06-01 12:17:49.219512 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30 2016-06-01 12:17:49.219

Re: [ceph-users] OSD Issue

2014-08-11 Thread Jacob Godin
Update #3: Our OSDs all crashed at the same time. Logs are all showing this: http://pastebin.com/ns0McteE On Mon, Aug 11, 2014 at 6:40 PM, Jacob Godin wrote: > We were able to get the cluster back online. The issue stemmed from the > MON having a lower epoch than the OSDs. > > We used ceph osd

Re: [ceph-users] OSD Issue

2014-08-11 Thread Jacob Godin
We were able to get the cluster back online. The issue stemmed from the MON having a lower epoch than the OSDs. We used ceph osd thrash to bring the MON's epoch up to be >= that of the OSDs, restarted the osd procs, and they began cooperating again. After they completed syncing, we're now running

[ceph-users] OSD Issue

2014-08-11 Thread Jacob Godin
Hi there, Currently having an issue with a Cuttlefish cluster w/ 3 OSDs and 1 MON. When trying to restart an OSD, the cluster became unresponsive to 'rbd export'. Here are some sample OSD logs: OSD we restarted -http://pastebin.com/UUuDdS1V Another OSD - http://pastebin.com/f12r4W2s In an attemp