What could cause this kind of unexpected behaviour? Any assumption?? Sorry for interrupting you.
Cheers, S On Tue, Mar 8, 2016 at 8:19 AM, Samuel Just <sj...@redhat.com> wrote: > Hmm, at the end of the log, the pg is still inconsistent. Can you > attach a ceph pg query on that pg? > -Sam > > On Mon, Mar 7, 2016 at 3:05 PM, Samuel Just <sj...@redhat.com> wrote: >> If so, that strongly suggests that the pg was actually never >> inconsistent in the first place and that the bug is in scrub itself >> presumably getting confused about an object during a write. The next >> step would be to get logs like the above from a pg as it scrubs >> transitioning from clean to inconsistent. If it's really a race >> between scrub and a write, it's probably just non-deterministic, you >> could set logging on a set of osds and continuously scrub any pgs >> which only map to those osds until you reproduce the problem. >> -Sam >> >> On Mon, Mar 7, 2016 at 2:44 PM, Samuel Just <sj...@redhat.com> wrote: >>> So after the scrub, it came up clean? The inconsistent/missing >>> objects reappeared? >>> -Sam >>> >>> On Mon, Mar 7, 2016 at 2:33 PM, Jeffrey McDonald <jmcdo...@umn.edu> wrote: >>>> Hi Sam, >>>> >>>> I've done as you requested: >>>> >>>> pg 70.459 is active+clean+inconsistent, acting [307,210,273,191,132,450] >>>> >>>> # for i in 307 210 273 191 132 450 ; do >>>>> ceph tell osd.$i injectargs '--debug-osd 20 --debug-filestore 20 >>>>> --debug-ms 1' >>>>> done >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1 >>>> >>>> >>>> # date >>>> Mon Mar 7 16:03:38 CST 2016 >>>> >>>> >>>> # ceph pg deep-scrub 70.459 >>>> instructing pg 70.459 on osd.307 to deep-scrub >>>> >>>> >>>> >>>> Scrub finished around >>>> >>>> # date >>>> Mon Mar 7 16:13:03 CST 2016 >>>> >>>> >>>> >>>> >>>> I've tar'd+gziped the files which can be downloaded from here. The logs >>>> start a minute or two after today at 16:00. >>>> >>>> https://drive.google.com/folderview?id=0Bzz8TrxFvfema2NQUmotd1BOTnM&usp=sharing >>>> >>>> >>>> Oddly(to me anyways), this pg is now active+clean: >>>> >>>> # ceph pg dump | grep 70.459 >>>> dumped all in format plain >>>> 70.459 21377 0 0 0 0 64515446306 3088 3088 active+clean 2016-03-07 >>>> 16:26:57.796537 279563'212832 279602:628151 [307,210,273,191,132,450] 307 >>>> [307,210,273,191,132,450] 307 279563'212832 2016-03-07 16:12:30.741984 >>>> 279563'212832 2016-03-07 16:12:30.741984 >>>> >>>> >>>> >>>> Regards, >>>> Jeff >>>> >>>> >>>> On Mon, Mar 7, 2016 at 4:11 PM, Samuel Just <sj...@redhat.com> wrote: >>>>> >>>>> I think the unfound object on repair is fixed by >>>>> d51806f5b330d5f112281fbb95ea6addf994324e (not in hammer yet). I >>>>> opened http://tracker.ceph.com/issues/15002 for the backport and to >>>>> make sure it's covered in ceph-qa-suite. No idea at this time why the >>>>> objects are disappearing though. >>>>> -Sam >>>>> >>>>> On Mon, Mar 7, 2016 at 1:57 PM, Samuel Just <sj...@redhat.com> wrote: >>>>> > The one just scrubbed and now inconsistent. >>>>> > -Sam >>>>> > >>>>> > On Mon, Mar 7, 2016 at 1:57 PM, Jeffrey McDonald <jmcdo...@umn.edu> >>>>> > wrote: >>>>> >> Do you want me to enable this for the pg already with unfound objects >>>>> >> or the >>>>> >> placement group just scrubbed and now inconsistent? >>>>> >> Jeff >>>>> >> >>>>> >> On Mon, Mar 7, 2016 at 3:54 PM, Samuel Just <sj...@redhat.com> wrote: >>>>> >>> >>>>> >>> Can you enable >>>>> >>> >>>>> >>> debug osd = 20 >>>>> >>> debug filestore = 20 >>>>> >>> debug ms = 1 >>>>> >>> >>>>> >>> on all osds in that PG, rescrub, and convey to us the resulting logs? >>>>> >>> -Sam >>>>> >>> >>>>> >>> On Mon, Mar 7, 2016 at 1:36 PM, Jeffrey McDonald <jmcdo...@umn.edu> >>>>> >>> wrote: >>>>> >>> > Here is a PG which just went inconsistent: >>>>> >>> > >>>>> >>> > pg 70.459 is active+clean+inconsistent, acting >>>>> >>> > [307,210,273,191,132,450] >>>>> >>> > >>>>> >>> > Attached is the result of a pg query on this. I will wait for your >>>>> >>> > feedback before issuing a repair. >>>>> >>> > >>>>> >>> > From what I read, the inconsistencies are more likely the result of >>>>> >>> > ntp, >>>>> >>> > but >>>>> >>> > all nodes have the local ntp master and all are showing sync. >>>>> >>> > >>>>> >>> > Regards, >>>>> >>> > Jeff >>>>> >>> > >>>>> >>> > On Mon, Mar 7, 2016 at 3:15 PM, Gregory Farnum <gfar...@redhat.com> >>>>> >>> > wrote: >>>>> >>> >> >>>>> >>> >> [ Keeping this on the users list. ] >>>>> >>> >> >>>>> >>> >> Okay, so next time this happens you probably want to do a pg query >>>>> >>> >> on >>>>> >>> >> the PG which has been reported as dirty. I can't help much beyond >>>>> >>> >> that, but hopefully Kefu or David will chime in once there's a >>>>> >>> >> little >>>>> >>> >> more for them to look at. >>>>> >>> >> -Greg >>>>> >>> >> >>>>> >>> >> On Mon, Mar 7, 2016 at 1:00 PM, Jeffrey McDonald <jmcdo...@umn.edu> >>>>> >>> >> wrote: >>>>> >>> >> > Hi Greg, >>>>> >>> >> > >>>>> >>> >> > I'm running the ceph version hammer, >>>>> >>> >> > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) >>>>> >>> >> > >>>>> >>> >> > The hardware migration was performed by just setting the crush >>>>> >>> >> > map to >>>>> >>> >> > zero >>>>> >>> >> > for the OSD we wanted to retire. The system was performing >>>>> >>> >> > poorly >>>>> >>> >> > with >>>>> >>> >> > these older OSDs and we had a difficult time maintaining >>>>> >>> >> > stability of >>>>> >>> >> > the >>>>> >>> >> > system. The old OSDs are still there but all of the data is >>>>> >>> >> > now >>>>> >>> >> > migrated >>>>> >>> >> > to new and/or existing hardware. >>>>> >>> >> > >>>>> >>> >> > Thanks, >>>>> >>> >> > Jeff >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > On Mon, Mar 7, 2016 at 2:56 PM, Gregory Farnum >>>>> >>> >> > <gfar...@redhat.com> >>>>> >>> >> > wrote: >>>>> >>> >> >> >>>>> >>> >> >> On Mon, Mar 7, 2016 at 12:07 PM, Jeffrey McDonald >>>>> >>> >> >> <jmcdo...@umn.edu> >>>>> >>> >> >> wrote: >>>>> >>> >> >> > Hi, >>>>> >>> >> >> > >>>>> >>> >> >> > For a while, we've been seeing inconsistent placement groups >>>>> >>> >> >> > on >>>>> >>> >> >> > our >>>>> >>> >> >> > erasure >>>>> >>> >> >> > coded system. The placement groups go from a state of >>>>> >>> >> >> > active+clean >>>>> >>> >> >> > to >>>>> >>> >> >> > active+clean+inconsistent after a deep scrub: >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > 2016-03-07 13:45:42.044131 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320s0 deep-scrub stat mismatch, got 21446/21428 objects, >>>>> >>> >> >> > 0/0 >>>>> >>> >> >> > clones, >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 >>>>> >>> >> >> > whiteouts, >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes. >>>>> >>> >> >> > 2016-03-07 13:45:42.044416 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320s0 deep-scrub 18 missing, 0 inconsistent objects >>>>> >>> >> >> > 2016-03-07 13:45:42.044464 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320 deep-scrub 73 errors >>>>> >>> >> >> > >>>>> >>> >> >> > So I tell the placement group to perform a repair: >>>>> >>> >> >> > >>>>> >>> >> >> > 2016-03-07 13:49:26.047177 7f385d118700 0 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [INF] : >>>>> >>> >> >> > 70.320 repair starts >>>>> >>> >> >> > 2016-03-07 13:49:57.087291 7f3858b0a700 0 -- >>>>> >>> >> >> > 10.31.0.2:6874/13937 >>>>> >>> >> >> > >> >>>>> >>> >> >> > 10.31.0.6:6824/8127 pipe(0x2e578000 sd=697 :6874 >>>>> >>> >> >> > >>>>> >>> >> >> > The repair finds missing shards and repairs them, but then I >>>>> >>> >> >> > have >>>>> >>> >> >> > 18 >>>>> >>> >> >> > 'unfound objects' : >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > 2016-03-07 13:51:28.467590 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320s0 repair stat mismatch, got 21446/21428 objects, 0/0 >>>>> >>> >> >> > clones, >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 >>>>> >>> >> >> > whiteouts, >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes. >>>>> >>> >> >> > 2016-03-07 13:51:28.468358 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320s0 repair 18 missing, 0 inconsistent objects >>>>> >>> >> >> > 2016-03-07 13:51:28.469431 7f385d118700 -1 >>>>> >>> >> >> > log_channel(cluster) >>>>> >>> >> >> > log >>>>> >>> >> >> > [ERR] : >>>>> >>> >> >> > 70.320 repair 73 errors, 73 fixed >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > I've traced one of the unfound objects all the way through the >>>>> >>> >> >> > system >>>>> >>> >> >> > and >>>>> >>> >> >> > I've found that they are not really lost. I can fail over >>>>> >>> >> >> > the >>>>> >>> >> >> > osd >>>>> >>> >> >> > and >>>>> >>> >> >> > recover the files. This is happening quite regularly now >>>>> >>> >> >> > after a >>>>> >>> >> >> > large >>>>> >>> >> >> > migration of data from old hardware to new(migration is now >>>>> >>> >> >> > complete). >>>>> >>> >> >> > >>>>> >>> >> >> > The system sets the PG into 'recovery', but we've seen the >>>>> >>> >> >> > system >>>>> >>> >> >> > in >>>>> >>> >> >> > a >>>>> >>> >> >> > recovering state for many days. Should we just be patient >>>>> >>> >> >> > or do >>>>> >>> >> >> > we >>>>> >>> >> >> > need >>>>> >>> >> >> > to dig further into the issue? >>>>> >>> >> >> >>>>> >>> >> >> You may need to dig into this more, although I'm not sure what >>>>> >>> >> >> the >>>>> >>> >> >> issue is likely to be. What version of Ceph are you running? How >>>>> >>> >> >> did >>>>> >>> >> >> you do this hardware migration? >>>>> >>> >> >> -Greg >>>>> >>> >> >> >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > pg 70.320 is stuck unclean for 704.803040, current state >>>>> >>> >> >> > active+recovering, >>>>> >>> >> >> > last acting [277,101,218,49,304,412] >>>>> >>> >> >> > pg 70.320 is active+recovering, acting >>>>> >>> >> >> > [277,101,218,49,304,412], >>>>> >>> >> >> > 18 >>>>> >>> >> >> > unfound >>>>> >>> >> >> > >>>>> >>> >> >> > There is no indication of any problems with down OSDs or >>>>> >>> >> >> > network >>>>> >>> >> >> > issues >>>>> >>> >> >> > with >>>>> >>> >> >> > OSDs. >>>>> >>> >> >> > >>>>> >>> >> >> > Thanks, >>>>> >>> >> >> > Jeff >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > -- >>>>> >>> >> >> > >>>>> >>> >> >> > Jeffrey McDonald, PhD >>>>> >>> >> >> > Assistant Director for HPC Operations >>>>> >>> >> >> > Minnesota Supercomputing Institute >>>>> >>> >> >> > University of Minnesota Twin Cities >>>>> >>> >> >> > 599 Walter Library email: >>>>> >>> >> >> > jeffrey.mcdon...@msi.umn.edu >>>>> >>> >> >> > 117 Pleasant St SE phone: +1 612 625-6905 >>>>> >>> >> >> > Minneapolis, MN 55455 fax: +1 612 624-8861 >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > >>>>> >>> >> >> > _______________________________________________ >>>>> >>> >> >> > ceph-users mailing list >>>>> >>> >> >> > ceph-users@lists.ceph.com >>>>> >>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>> >> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> >> > -- >>>>> >>> >> > >>>>> >>> >> > Jeffrey McDonald, PhD >>>>> >>> >> > Assistant Director for HPC Operations >>>>> >>> >> > Minnesota Supercomputing Institute >>>>> >>> >> > University of Minnesota Twin Cities >>>>> >>> >> > 599 Walter Library email: jeffrey.mcdon...@msi.umn.edu >>>>> >>> >> > 117 Pleasant St SE phone: +1 612 625-6905 >>>>> >>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861 >>>>> >>> >> > >>>>> >>> >> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > -- >>>>> >>> > >>>>> >>> > Jeffrey McDonald, PhD >>>>> >>> > Assistant Director for HPC Operations >>>>> >>> > Minnesota Supercomputing Institute >>>>> >>> > University of Minnesota Twin Cities >>>>> >>> > 599 Walter Library email: jeffrey.mcdon...@msi.umn.edu >>>>> >>> > 117 Pleasant St SE phone: +1 612 625-6905 >>>>> >>> > Minneapolis, MN 55455 fax: +1 612 624-8861 >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > _______________________________________________ >>>>> >>> > ceph-users mailing list >>>>> >>> > ceph-users@lists.ceph.com >>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>> > >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> >>>>> >> Jeffrey McDonald, PhD >>>>> >> Assistant Director for HPC Operations >>>>> >> Minnesota Supercomputing Institute >>>>> >> University of Minnesota Twin Cities >>>>> >> 599 Walter Library email: jeffrey.mcdon...@msi.umn.edu >>>>> >> 117 Pleasant St SE phone: +1 612 625-6905 >>>>> >> Minneapolis, MN 55455 fax: +1 612 624-8861 >>>>> >> >>>>> >> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Jeffrey McDonald, PhD >>>> Assistant Director for HPC Operations >>>> Minnesota Supercomputing Institute >>>> University of Minnesota Twin Cities >>>> 599 Walter Library email: jeffrey.mcdon...@msi.umn.edu >>>> 117 Pleasant St SE phone: +1 612 625-6905 >>>> Minneapolis, MN 55455 fax: +1 612 624-8861 >>>> >>>> > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Email: shin...@linux.com GitHub: shinobu-x Blog: Life with Distributed Computational System based on OpenSource _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
