Re: [ceph-users] Question about PGMonitor::waiting_for_finished_proposal

2017-06-01 Thread Joao Eduardo Luis
On 06/01/2017 05:35 AM, 许雪寒 wrote: Hi, everyone. Recently, I’m reading the source code of Monitor. I found that, in PGMonitor::preprare_pg_stats() method, a callback C_Stats is put into PGMonitor::waiting_for_finished_proposal. I wonder, if a previous PGMap incremental is in PAXOS's propose

[ceph-users] RGW: Truncated objects and bad error handling

2017-06-01 Thread Jens Rosenboom
On a large Hammer-based cluster (> 1 Gobjects) we are seeing a small amount of objects being truncated. All of these objects are between 512kB and 4MB in size and they are not uploaded as multipart, so the first 512kB get stored into the head object and the next chunks should be in tail objects nam

[ceph-users] PG Stuck EC Pool

2017-06-01 Thread Ashley Merrick
Have a PG which is stuck in this state (Is an EC with K=10 M=3) pg 6.14 is active+undersized+degraded+remapped+inconsistent+backfilling, acting [2147483647,2147483647,84,83,22,26,69,72,53,59,8,4,46] Currently have no-recover set, if I unset no recover both OSD 83 + 84 start to flap and go up

Re: [ceph-users] rbd map fails, ceph release jewel

2017-06-01 Thread Shambhu Rajak
Thanks David, I upgraded the kernel version and the rbd map worked. Regards, Shambhu From: David Turner [mailto:drakonst...@gmail.com] Sent: Wednesday, May 31, 2017 9:35 PM To: Shambhu Rajak; ceph-users@lists.ceph.com Subject: Re: [ceph-users] rbd map fails, ceph release jewel You are trying to u

[ceph-users] Read errors on OSD

2017-06-01 Thread Oliver Humpage
Hello, We have a small cluster of 44 OSDs across 4 servers. A few times a week, ceph health reports a pg is inconsistent. Looking at the relevant OSD’s logs, it always says "head candidate had a read error”. No other info, i.e. it’s not that the digest is wrong, it just has an I/O error. It’s

Re: [ceph-users] Read errors on OSD

2017-06-01 Thread Matthew Vernon
Hi, On 01/06/17 10:38, Oliver Humpage wrote: These read errors are all on Samsung 850 Pro 2TB disks (journals are on separate enterprise SSDs). The SMART status on all of them are similar and show nothing out of the ordinary. Has anyone else experienced anything similar? Is this just a curse o

Re: [ceph-users] Read errors on OSD

2017-06-01 Thread Oliver Humpage
> On 1 Jun 2017, at 11:55, Matthew Vernon wrote: > > You don't say what's in kern.log - we've had (rotating) disks that were > throwing read errors but still saying they were OK on SMART. Fair point. There was nothing correlating to the time that ceph logged an error this morning, which is wh

[ceph-users] tools to display information from ceph report

2017-06-01 Thread Loic Dachary
Hi, Is there a tool that displays information (such as the total bytes in each pool) using the content of the "ceph report" json ? Cheers -- Loïc Dachary, Artisan Logiciel Libre ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.

Re: [ceph-users] http://planet.eph.com/ is down

2017-06-01 Thread Patrick McGarry
Hey Loic, I have updated planet.ceph.com with a 301 that redirects to the /category/planet, so I think it should be set now. Thanks. On Sun, May 28, 2017 at 1:29 AM, Loic Dachary wrote: > The URL is http://ceph.com/category/planet/ and works like a charm :-) There > is a blog at http://eph.com

Re: [ceph-users] Read errors on OSD

2017-06-01 Thread Steve Taylor
I've seen similar issues in the past with 4U Supermicro servers populated with spinning disks. In my case it turned out to be a specific firmware+BIOS combination on the disk controller card that was buggy. I fixed it by updating the firmware and BIOS on the card to the latest versions. I saw t

[ceph-users] RBD exclusive-lock and lqemu/librbd

2017-06-01 Thread koukou73gr
Hello list, Today I had to create a new image for a VM. This was the first time, since our cluster was updated from Hammer to Jewel. So far I was just copying an existing golden image and resized it as appropriate. But this time I used rbd create. So I "rbd create"d a 2T image and attached it to

[ceph-users] Editing Ceph source code and debugging

2017-06-01 Thread Oleg Kolosov
Hi I'm interested in writing an original erasure code in Ceph for my research purposes. I was wondering if there is any tool or method supporting quick compilation and debugging. Thanks ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.c

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

2017-06-01 Thread Jake Grimmett
Hi Mark, Firstly, many thanks for looking into this. Jayaram appears to have a similar config to me; v12.0.3, EC 4+1 bluestore SciLin,7.3 - 3.10.0-514.21.1.el7.x86_64 I have 5 EC nodes (10 x 8TB ironwolf each) plus 2 nodes with replicated NVMe (Cephfs hot tier) I now think the Highpoint r750 r

Re: [ceph-users] PG Stuck EC Pool

2017-06-01 Thread Ashley Merrick
Have attached the full pg query for the effected PG encase this shows anything of interest. Thanks From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ashley Merrick Sent: 01 June 2017 17:19 To: ceph-us...@ceph.com Subject: [ceph-users] PG Stuck EC Pool This sender failed

[ceph-users] Crushmap from Rack aware to Node aware

2017-06-01 Thread Deepak Naidu
Greetings Folks. Wanted to understand how ceph works when we start with rack aware(rack level replica) example 3 racks and 3 replica in crushmap in future is replaced by node aware(node level replica) ie 3 replica spread across nodes. This can be vice-versa. If this happens. How does ceph rearr

Re: [ceph-users] Editing Ceph source code and debugging

2017-06-01 Thread David Turner
I'm pretty sure this is a question for ceph-devel. On Thu, Jun 1, 2017 at 11:22 AM Oleg Kolosov wrote: > Hi > I'm interested in writing an original erasure code in Ceph for my research > purposes. I was wondering if there is any tool or method supporting quick > compilation and debugging. > > Th

Re: [ceph-users] Read errors on OSD

2017-06-01 Thread Oliver Humpage
> On 1 Jun 2017, at 14:38, Steve Taylor wrote: > > I saw this on several servers, and it took a while to track down as you can > imagine. Same symptoms you're reporting. Thanks, that’s very useful info. We’re using separate Adaptec controllers, but will double check firmware on them. Who know

Re: [ceph-users] RGW: Truncated objects and bad error handling

2017-06-01 Thread Gregory Farnum
On Thu, Jun 1, 2017 at 2:03 AM Jens Rosenboom wrote: > On a large Hammer-based cluster (> 1 Gobjects) we are seeing a small > amount of objects being truncated. All of these objects are between > 512kB and 4MB in size and they are not uploaded as multipart, so the > first 512kB get stored into th

Re: [ceph-users] Crushmap from Rack aware to Node aware

2017-06-01 Thread David Turner
The way to do this is to download your crush map, modify it manually after decompiling it to text format or modify it using the crushtool. Once you have your crush map with the rules in place that you want, you will upload the crush map to the cluster. When you change your failure domain from hos

Re: [ceph-users] Crushmap from Rack aware to Node aware

2017-06-01 Thread Deepak Naidu
Perfect David for detailed explanation. Appreciate it!. In my case I have 10 OSD servers with each 60 Disks(ya I know…) ie total 600 OSD and I have 3 racks to spare. -- Deepak From: David Turner [mailto:drakonst...@gmail.com] Sent: Thursday, June 01, 2017 12:23 PM To: Deepak Naidu; ceph-users S

Re: [ceph-users] Crushmap from Rack aware to Node aware

2017-06-01 Thread David Turner
If all 6 racks are tagged for Ceph storage nodes, I'd go ahead and just put the nodes in there now and configure the crush map accordingly. That way you can grow each of the racks while keeping each failure domain closer in size to the rest of the cluster. On Thu, Jun 1, 2017 at 3:40 PM Deepak Na

Re: [ceph-users] Crushmap from Rack aware to Node aware

2017-06-01 Thread Deepak Naidu
>> If all 6 racks are tagged for Ceph storage nodes, I'd go ahead and just put >> the nodes in there now and configure the crush map accordingly I just have 3 racks. That’s the max I have for now. 10 OSD Nodes. -- Deepak From: David Turner [mailto:drakonst...@gmail.com] Sent: Thursday, June 01,

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

2017-06-01 Thread Mark Nelson
Looking at this gdb output, it looks like all of the tp_osd_tp threads are idling around except for three that are all waiting on a PG lock. I bet those sit there for 60s and eventually time out. The kv_sync_thread looks idle so I don't think that's it. Thread 16 is doing OSD::trim_maps but

[ceph-users] is there any way to speed up cache evicting?

2017-06-01 Thread jiajia zhong
hi guys: Our ceph cluster is working with tier cache. I am running "rados -p data_cache cache-try-flush-evict-all" to evict all the objects. But It a bit slow 1. Is there any way to speed up the evicting? 2. Is evicting triggered by itself good enough for cluster ? 3. Does the flushing and evic

Re: [ceph-users] ceph packages on stretch from eu.ceph.com

2017-06-01 Thread Christian Balzer
Hello, Sorry for the thread necromancy. With Stretch deep-frozen and amazingly enough on schedule for release in 2 weeks (and me having to finish a new cluster deployment by July), I sure hope that whoever is in charge of this has everything set up and just needs to push a button for things to b

Re: [ceph-users] is there any way to speed up cache evicting?

2017-06-01 Thread Christian Balzer
On Fri, 2 Jun 2017 10:30:46 +0800 jiajia zhong wrote: > hi guys: > > Our ceph cluster is working with tier cache. If so, then I suppose you read all the discussions here as well and not only the somewhat lacking documentation? > I am running "rados -p data_cache cache-try-flush-evict-all" to evi

Re: [ceph-users] is there any way to speed up cache evicting?

2017-06-01 Thread jiajia zhong
christian, thanks for your reply. 2017-06-02 11:39 GMT+08:00 Christian Balzer : > On Fri, 2 Jun 2017 10:30:46 +0800 jiajia zhong wrote: > > > hi guys: > > > > Our ceph cluster is working with tier cache. > If so, then I suppose you read all the discussions here as well and not > only the somewhat