Hi,
I think I found what trigger mon reelection.
It is system disk load (logrotate, and other ...), I have mon
directory on this disk aslo.
Regards
Dominik
2013/9/13 Gregory Farnum :
> I believe that's too high of an allowed skew with the default lease etc
> settings. The actual complaint is "I got a lease which has ALREADY expired
> and can't do anything with that!"
>
> You'll need to either get your clock skew down to less than, say, 1/4 second
> (which is perfectly doable over three nodes with ntp), or go through and
> change all of the monitor timing configurables appropriately for that skew.
> I don't remember all the constraints you'll need to satisfy when doing that
> so I really recommend the first option.
> -Greg
>
> On Friday, September 13, 2013, Dominik Mostowiec wrote:
>>
>> Hi,
>> I have ntpd installed on servers, time seems to be ok.
>>
>> I have strange log:
>> 2013-09-12 07:34:40.238659 7fd63ac3e700 -1
>> mon.4@3(peon).p0.075434axos(auth active c 581328..581348) lease_expire
>> from mon.0 10.177.64.4:6789/0 is seconds in the past; mons are laggy
>> or clocks are too skewed
>>
>> But value 0.075434 is small.
>>
>> In ceph.conf i have:
>> mon allowed clock drift = 2
>>
>> Some time mons reports:
>> 2013-09-13 00:11:14.556410 7fd63ac3e700 0 log [INF] : mon.4 calling
>> new monitor election
>> 2013-09-13 00:11:14.557306 7fd6317b8700 0 -- 10.177.64.7:6789/0 >>
>> 10.177.64.5:6789/0 pipe(0xdbc2000 sd=18 :6789 s=0 pgs=0 cs=0
>> l=0).accept connect_seq 112 vs existing 112 state connecting
>> 2013-09-13 00:11:14.557374 7fd638525700 0 -- 10.177.64.7:6789/0 >>
>> 10.177.64.9:6789/0 pipe(0x14766c80 sd=24 :6789 s=0 pgs=0 cs=0
>> l=0).accept connect_seq 112 vs existing 112 state connecting
>> 2013-09-13 00:11:14.557398 7fd638c2c700 0 -- 10.177.64.7:6789/0 >>
>> 10.177.64.6:6789/0 pipe(0x14766a00 sd=23 :6789 s=0 pgs=0 cs=0
>> l=0).accept connect_seq 126 vs existing 126 state connecting
>> 2013-09-13 00:11:16.467636 7fd631ebf700 0 -- 10.177.64.7:6789/0 >>
>> 10.177.64.8:6789/0 pipe(0x7038c80 sd=20 :6789 s=0 pgs=0 cs=0
>> l=0).accept connect_seq 122 vs existing 122 state connecting
>> 2013-09-13 00:11:21.553559 7fd63ac3e700 0 log [INF] : mon.4 calling
>> new monitor election
>>
>>
>> --
>> Dominik
>>
>> 2013/9/13 Joao Eduardo Luis :
>> > On 09/13/2013 03:38 AM, Sage Weil wrote:
>> >>
>> >> On Thu, 12 Sep 2013, Dominik Mostowiec wrote:
>> >>>
>> >>> Hi,
>> >>> Today i have some issues with ceph cluster.
>> >>> After new mon election many osd has been marked failed.
>> >>> Some time later osd boot and i think recover because meny slow request
>> >>> appear.
>> >>> Cluster come back after about 20minutes.
>> >>
>> >>
>> >> Was there some other event that triggered the mon election? There's
>> >> not
>> >> much here to go on except that several elections were called and by
>> >> different monitors, which suggests something was not quite right.
>> >
>> >
>> > My best guess would be clock skews, as they tend to be annoying like
>> > that.
>> >
>> > Setting 'debug mon = 10' on the monitors should provide more insight
>> > though.
>> >
>> > -Joao
>> >
>> >>
>> >> sage
>> >>
>> >>
>> >>>
>> >>> cluster:
>> >>> ceph version 0.56.6
>> >>> 6 servers x 26 osd
>> >>>
>> >>> 2013-09-12 07:11:40.920384 mon.1 10.177.64.5:6789/0 353 : [INF] mon.3
>> >>> calling new monitor election
>> >>> 2013-09-12 07:12:40.992532 mon.3 10.177.64.7:6789/0 364 : [INF] mon.4
>> >>> calling new monitor election
>> >>> 2013-09-12 07:12:41.024954 mon.4 10.177.64.8:6789/0 360 : [INF] mon.2
>> >>> calling new monitor election
>> >>> 2013-09-12 07:13:02.782203 mon.2 10.177.64.6:6789/0 336 : [INF] mon.1
>> >>> calling new monitor election
>> >>> 2013-09-12 07:13:02.783778 mon.3 10.177.64.7:6789/0 366 : [INF] mon.4
>> >>> calling new monitor election
>> >>> 2013-09-12 07:13:10.852842 mon.3 10.177.64.7:6789/0 367 : [INF] mon.4
>> >>> calling new monitor election
>> >>> 2013-09-12 16:17:09.484277 mon.4 10.177.64.8:6789/0 363 : [INF] mon.2
>> >>> calling new monitor election
>> >>> 2013-09-12 16:17:09.497337 mon.3 10.177.64.7:6789/0 368 : [INF] mon.4
>> >>> calling new monitor election
>> >>> 2013-09-12 16:17:09.523787 mon.0 10.177.64.4:6789/0 4369021 : [INF]
>> >>> mon.0 calling new monitor election
>> >>> 2013-09-12 16:17:14.525282 mon.0 10.177.64.4:6789/0 4369022 : [INF]
>> >>> mon.0@0 won leader election with quorum 0,1,2,3,4
>> >>> ...
>> >>> 2013-09-12 16:17:14.689555 mon.0 10.177.64.4:6789/0 4369027 : [DBG]
>> >>> osd.130 10.177.64.9:6801/1401 reported failed by osd.121
>> >>> 10.177.64.7:6909/29496
>> >>> 2013-09-12 16:17:14.689584 mon.0 10.177.64.4:6789/0 4369028 : [DBG]
>> >>> osd.131 10.177.64.9:6810/2435 reported failed by osd.121
>> >>> 10.177.64.7:6909/29496
>> >>> 2013-09-12 16:17:14.689600 mon.0 10.177.64.4:6789/0 4369029 : [DBG]
>> >>> osd.132 10.177.64.9:6846/2885 reported failed by osd.121
>> >>> 10.177.64.7:6909/29496
>> >>> 2013-09-12 16:17:14.689615 mon.0 10.177.64.4:6789/0 4369030 : [DBG]
>> >>> osd.134 10.177.
