Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Sage Weil
On Thu, 5 Feb 2015, Andrey Korolyov wrote: > > > > Yep, it's a silly bug and I'm surprised we haven't noticed until now! > > > > http://tracker.ceph.com/issues/10762 > > https://github.com/ceph/ceph/pull/3631 > > > > Thanks! > > sage > > Thanks Sage, is dumpling missing from backpo

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Andrey Korolyov
> > Yep, it's a silly bug and I'm surprised we haven't noticed until now! > > http://tracker.ceph.com/issues/10762 > https://github.com/ceph/ceph/pull/3631 > > Thanks! > sage Thanks Sage, is dumpling missing from backport list by a purpose? _

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Sage Weil
On Thu, 5 Feb 2015, Dan van der Ster wrote: > On Thu, Feb 5, 2015 at 9:54 AM, Sage Weil wrote: > > On Thu, 5 Feb 2015, Dan van der Ster wrote: > >> Hi, > >> We also have seen this once after upgrading to 0.80.8 (from dumpling). > >> Last week we had a network outage which marked out around 1/3rd o

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Dan van der Ster
On Thu, Feb 5, 2015 at 9:54 AM, Sage Weil wrote: > On Thu, 5 Feb 2015, Dan van der Ster wrote: >> Hi, >> We also have seen this once after upgrading to 0.80.8 (from dumpling). >> Last week we had a network outage which marked out around 1/3rd of our >> OSDs. The outage lasted less than a minute --

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Christian Eichelmann
Am 05.02.2015 10:10, schrieb Dan van der Ster: > > But then when I restarted the (peon) monitor: > > 2015-01-29 11:29:18.250750 mon.0 128.142.35.220:6789/0 10570 : [INF] > pgmap v35847068: 24608 pgs: 1 active+clean+scrubbing+deep, 24602 > active+clean, 5 active+clean+scrubbing; 125 T > B data, 37

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Sahana Lokeshappa
Dan van der Ster Sent: Thursday, February 05, 2015 2:41 PM To: Sage Weil Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down On Thu, Feb 5, 2015 at 9:54 AM, Sage Weil wrote: > On Thu, 5 Feb 2015, Dan van der Ster wrote: >> Hi, >

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Dan van der Ster
On Thu, Feb 5, 2015 at 9:54 AM, Sage Weil wrote: > On Thu, 5 Feb 2015, Dan van der Ster wrote: >> Hi, >> We also have seen this once after upgrading to 0.80.8 (from dumpling). >> Last week we had a network outage which marked out around 1/3rd of our >> OSDs. The outage lasted less than a minute --

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Sage Weil
On Thu, 5 Feb 2015, Dan van der Ster wrote: > Hi, > We also have seen this once after upgrading to 0.80.8 (from dumpling). > Last week we had a network outage which marked out around 1/3rd of our > OSDs. The outage lasted less than a minute -- all the OSDs were > brought up once the network was res

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-05 Thread Dan van der Ster
Hi, We also have seen this once after upgrading to 0.80.8 (from dumpling). Last week we had a network outage which marked out around 1/3rd of our OSDs. The outage lasted less than a minute -- all the OSDs were brought up once the network was restored. Then 30 minutes later I restarted one monitor

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-04 Thread Christian Eichelmann
Hi Greg, the behaviour is indeed strange. Today I was trying to reproduce the problem, but no matter which monitor I've restarted, no matter how many times, the bahviour was like expected: A new monitor election was called and everything contiuned normally. Then I continued my failover tests and

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-03 Thread Gregory Farnum
On Tue, Feb 3, 2015 at 3:38 AM, Christian Eichelmann wrote: > Hi all, > > during some failover tests and some configuration tests, we currently > discover a strange phenomenon: > > Restarting one of our monitors (5 in sum) triggers about 300 of the > following events: > > osd.669 10.76.28.58:6935/

Re: [ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-03 Thread Andrey Korolyov
On Tue, Feb 3, 2015 at 2:38 PM, Christian Eichelmann wrote: > Hi all, > > during some failover tests and some configuration tests, we currently > discover a strange phenomenon: > > Restarting one of our monitors (5 in sum) triggers about 300 of the > following events: > > osd.669 10.76.28.58:6935/

[ceph-users] Monitor Restart triggers half of our OSDs marked down

2015-02-03 Thread Christian Eichelmann
Hi all, during some failover tests and some configuration tests, we currently discover a strange phenomenon: Restarting one of our monitors (5 in sum) triggers about 300 of the following events: osd.669 10.76.28.58:6935/149172 failed (20 reports from 20 peers after 22.005858 >= grace 20.00)