Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
Well since I spammed the list earlier, I should fess up to my mistakes. I forgot to change MTU sizes on the 10G switch after I switched to jumbo frames. So yes, I had a very unhappy networking stack. On the upside, playing with Cumulus Linux on switches is fun. On Tue, Mar 25, 2014 at 1:12 PM,

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
Thanks for the feedback -- I'll post back with more detailed logs if anything looks fishy! On Tue, Mar 25, 2014 at 1:10 PM, Gregory Farnum wrote: > Well, you could try running with messenger debugging cranked all the > way up and see if there's something odd happening there (eg, not > handling

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Gregory Farnum
Well, you could try running with messenger debugging cranked all the way up and see if there's something odd happening there (eg, not handling incoming messages), but based on not having any other reports of this, I think your networking stack is unhappy in some way. *shrug* (Higher log levels show

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
On Tue, Mar 25, 2014 at 12:53 PM, Gregory Farnum wrote: > On Tue, Mar 25, 2014 at 9:24 AM, Travis Rhoden wrote: > > Okay, last one until I get some guidance. Sorry for the spam, but > wanted to > > paint a full picture. Here are debug logs from all three mons, capturing > > what looks like an

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Gregory Farnum
On Tue, Mar 25, 2014 at 9:24 AM, Travis Rhoden wrote: > Okay, last one until I get some guidance. Sorry for the spam, but wanted to > paint a full picture. Here are debug logs from all three mons, capturing > what looks like an election sequence to me: > > ceph0: > 2014-03-25 16:17:24.324846 7fa

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
> De: "Travis Rhoden" > À: "ceph-users" > Envoyé: Mardi 25 Mars 2014 17:24:19 > Objet: Re: [ceph-users] Monitors stuck in "electing" > > > > > > Okay, last one until I get some guidance. Sorry for the spam, but wanted > to paint a ful

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Alexandre DERUMIER
Hi, can you ping between your hosts ? (just to be sure, what is your netmask ? (as I see 10.10.30.0 for mon1) - Mail original - De: "Travis Rhoden" À: "ceph-users" Envoyé: Mardi 25 Mars 2014 17:24:19 Objet: Re: [ceph-users] Monitors stuck in "electing&quo

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
Okay, last one until I get some guidance. Sorry for the spam, but wanted to paint a full picture. Here are debug logs from all three mons, capturing what looks like an election sequence to me: ceph0: 2014-03-25 16:17:24.324846 7fa5c53fc700 5 mon.ceph0@0(electing).elector(35) start -- can i be l

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
I bumped debug mon and debug ms up on one of the monitors (ceph0), and this is what I see: 2014-03-25 16:02:19.273406 7fa5c53fc700 5 mon.ceph0@0(electing).elector(35) election timer expired 2014-03-25 16:02:19.273447 7fa5c53fc700 5 mon.ceph0@0(electing).elector(35) start -- can i be leader? 2014

Re: [ceph-users] Monitors stuck in "electing"

2014-03-25 Thread Travis Rhoden
Just to emphasize that I don't think it's clock skew, here is the NTP state of all three monitors: # ansible ceph_mons -m command -a "ntpq -p" -kK SSH password: sudo password [defaults to SSH password]: ceph0 | success | rc=0 >> remote refid st t when poll reach delay offse