Re: [ceph-users] scrub error on firefly

2014-07-15 Thread Sage Weil
> >> >> >>> >> >> > be > >> >> >>> >> >> > "close" > >> >> >>> >> >> > to the previ

Re: [ceph-users] scrub error on firefly

2014-07-12 Thread Samuel Just
gt; >> > Greetings, >> >> >>> >> >> > >> >> >>> >> >> > Just a follow up on my original issue. =ceph pg repair ...= >> >> >>> >> >> > fixed >> >> >>> >> >>

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Randy Smith
gt;> > different.) > >>> >> >> > > >>> >> >> > Is this indicative of a problem on osd.5 or perhaps a clue into > >>> >> >> > what's > >>> >> >> > causing firefly to be so inconsistent? > >

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Samuel Just
sd.2 192.168.253.70:6801/56987 163 : >>> >> >> > [ERR] >>> >> >> > 3.c6 >>> >> >> > shard 2: soid 34dc35c6/rb.0.b0ce3.238e1f29.000b/head//3 >>> >> >> > digest >>> >> >> >

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Samuel Just
8068918 >> >> >> > 2014-07-07 18:51:36.936076 osd.2 192.168.253.70:6801/56987 164 : >> >> >> > [ERR] >> >> >> > 3.c6 >> >> >> > deep-scrub 0 missing, 1 inconsistent objects >> >> >> > 2014-07-07 18:51:36.93

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Randy Smith
; >> >> > 3.c6 > >> >> > deep-scrub 1 errors > >> >> > > >> >> > > >> >> > 2014-07-10 15:38:53.990328 osd.5 192.168.253.81:6800/10013 257 : > [ERR] > >> >> > 3.41 > >> >> >

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Samuel Just
.0.b0ce3.238e1f29.024c/head//3 digest >> >> > 3224286363 != known digest 3409342281 >> >> > 2014-07-10 15:39:11.701276 osd.5 192.168.253.81:6800/10013 258 : [ERR] >> >> > 3.41 >> >> > deep-scrub 0 missing, 1 inconsistent objects >> &g

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Sage Weil
gt; > 2014-07-10 15:39:11.701281 osd.5 192.168.253.81:6800/10013 259 : [ERR] > >> > 3.41 > >> > deep-scrub 1 errors > >> > > >> > > >> > > >> > On Thu, Jul 10, 2014 at 12:05 PM, Chahal, Sudip > >> > wrote: &

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Samuel Just
; > any insights: >> >> > >> >> > With three replicas, if checksum inconsistencies across replicas are >> >> > found during deep-scrub then: >> >> > a. does the majority win or is the primary always the winner >> >> > an

Re: [ceph-users] scrub error on firefly

2014-07-11 Thread Randy Smith
istencies found during deep scrub - would you > >> agree? > >> > >> Re: repair - do you mean the "repair" process during deep scrub - if > yes, > >> this is automatic - correct? > >> Or > >> Are you referring to the explicit ma

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Samuel Just
manually initiated repair commands? >> >> Thanks, >> >> -Sudip >> >> -Original Message- >> From: Samuel Just [mailto:sam.j...@inktank.com] >> Sent: Thursday, July 10, 2014 10:50 AM >> To: Chahal, Sudip >> Cc: Christian Eichelmann; cep

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Randy Smith
t; Or > Are you referring to the explicit manually initiated repair commands? > > Thanks, > > -Sudip > > -Original Message- > From: Samuel Just [mailto:sam.j...@inktank.com] > Sent: Thursday, July 10, 2014 10:50 AM > To: Chahal, Sudip > Cc: Christian E

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Chahal, Sudip
mann; ceph-users@lists.ceph.com Subject: Re: [ceph-users] scrub error on firefly Repair I think will tend to choose the copy with the lowest osd number which is not obviously corrupted. Even with three replicas, it does not do any kind of voting at this time. -Sam On Thu, Jul 10, 2014 at 10:39

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Samuel Just
On Behalf Of > Samuel Just > Sent: Thursday, July 10, 2014 10:16 AM > To: Christian Eichelmann > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] scrub error on firefly > > Can you attach your ceph.conf for your osds? > -Sam > > On Thu, Jul 10, 2014 at 8:01

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Chahal, Sudip
014 10:16 AM To: Christian Eichelmann Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] scrub error on firefly Can you attach your ceph.conf for your osds? -Sam On Thu, Jul 10, 2014 at 8:01 AM, Christian Eichelmann wrote: > I can also confirm that after upgrading to firefly both of our

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Samuel Just
"Travis > Rhoden [trho...@gmail.com] > Gesendet: Donnerstag, 10. Juli 2014 16:24 > An: Gregory Farnum > Cc: ceph-users@lists.ceph.com > Betreff: Re: [ceph-users] scrub error on firefly > > And actually just to follow-up, it does seem like there are some additional >

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Christian Eichelmann
Christian Von: ceph-users [ceph-users-boun...@lists.ceph.com]" im Auftrag von "Travis Rhoden [trho...@gmail.com] Gesendet: Donnerstag, 10. Juli 2014 16:24 An: Gregory Farnum Cc: ceph-users@lists.ceph.com Betreff: Re: [ceph-users] scrub error on firefly And actually just t

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Travis Rhoden
And actually just to follow-up, it does seem like there are some additional smarts beyond just using the primary to overwrite the secondaries... Since I captured md5 sums before and after the repair, I can say that in this particular instance, the secondary copy was used to overwrite the primary.

Re: [ceph-users] scrub error on firefly

2014-07-10 Thread Travis Rhoden
I can also say that after a recent upgrade to Firefly, I have experienced massive uptick in scrub errors. The cluster was on cuttlefish for about a year, and had maybe one or two scrub errors. After upgrading to Firefly, we've probably seen 3 to 4 dozen in the last month or so (was getting 2-3 a

Re: [ceph-users] scrub error on firefly

2014-07-07 Thread Gregory Farnum
It's not very intuitive or easy to look at right now (there are plans from the recent developer summit to improve things), but the central log should have output about exactly what objects are busted. You'll then want to compare the copies manually to determine which ones are good or bad, get the g

[ceph-users] scrub error on firefly

2014-07-07 Thread Randy Smith
Greetings, I upgraded to firefly last week and I suddenly received this error: health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors ceph health detail shows the following: HEALTH_ERR 1 pgs inconsistent; 1 scrub errors pg 3.c6 is active+clean+inconsistent, acting [2,5] 1 scrub errors The docs s