Re: [ceph-users] ceph pg query hangs for ever

2016-04-01 Thread Florian Haas
On Fri, Apr 1, 2016 at 2:48 PM, Wido den Hollander wrote: > Somehow the PG got corrupted on one of the OSDs and it kept crashing on a > single > object. Vaguely reminds me of the E2BIG from that one issue way-back-when in Dumpling (https://www.hastexo.com/resources/hints-and-kinks/fun-extended

Re: [ceph-users] ceph pg query hangs for ever

2016-04-01 Thread Wido den Hollander
> Op 1 april 2016 om 1:28 schreef Goncalo Borges : > > > Hi Mart, Wido... > > A disclaimer: Not really an expert, just a regular site admin sharing my > experience. > Thanks! > At the beginning of the thread you give the idea that only osd.68 has > problems dealing with the problematic PG

Re: [ceph-users] ceph pg query hangs for ever

2016-03-31 Thread Goncalo Borges
Hi Mart, Wido... A disclaimer: Not really an expert, just a regular site admin sharing my experience. At the beginning of the thread you give the idea that only osd.68 has problems dealing with the problematic PG 3.117. If that is indeed the case, you could simply mark that osd.68 down and r

Re: [ceph-users] ceph pg query hangs for ever

2016-03-30 Thread Mart van Santen
Hello, Well unfortunately the problem is not really solved. Yes, we managed to get to a good health state at some point, when a client hits some specific data, the osd process crashes with below errors. The 3 OSD which handle 3.117, the PG with problems, are currently down and reweighted them to

Re: [ceph-users] ceph pg query hangs for ever

2016-03-30 Thread Mart van Santen
Hi there, With the help of a lot of people we were able to repair the PG and restored service. We will get back on this later with a full report for future reference. Regards, Mart On 03/30/2016 08:30 PM, Wido den Hollander wrote: > Hi, > > I have an issue with a Ceph cluster which I can't re