Re: [ceph-users] Much more dentries than inodes, is that normal?

2017-03-09 Thread Xiaoxi Chen
Yeah I checked the dump , it it truely the known issue. Thanks 2017-03-08 17:58 GMT+08:00 John Spray : > On Tue, Mar 7, 2017 at 3:05 PM, Xiaoxi Chen wrote: >> Thanks John. >> >> Very likely, note that mds_mem::ino + mds_cache::strays_created ~= >> mds::inodes, plus the MDS was the active-standb

[ceph-users] RGW listing users' quota and usage painfully slow

2017-03-09 Thread Matthew Vernon
Hi, I'm using Jewel / 10.2.3-0ubuntu0.16.04.2 . We want to keep track of our S3 users' quota and usage. Even with a relatively small number of users (23) it's taking ~23 seconds. What we do is (in outline): radosgw-admin metadata list user for each user X: radosgw-admin user info --uid=X #

Re: [ceph-users] RGW listing users' quota and usage painfully slow

2017-03-09 Thread Abhishek Lekshmanan
On 03/09/2017 11:26 AM, Matthew Vernon wrote: Hi, I'm using Jewel / 10.2.3-0ubuntu0.16.04.2 . We want to keep track of our S3 users' quota and usage. Even with a relatively small number of users (23) it's taking ~23 seconds. What we do is (in outline): radosgw-admin metadata list user for eac

Re: [ceph-users] RGW listing users' quota and usage painfully slow

2017-03-09 Thread Matthew Vernon
On 09/03/17 10:45, Abhishek Lekshmanan wrote: On 03/09/2017 11:26 AM, Matthew Vernon wrote: I'm using Jewel / 10.2.3-0ubuntu0.16.04.2 . We want to keep track of our S3 users' quota and usage. Even with a relatively small number of users (23) it's taking ~23 seconds. What we do is (in outline)

Re: [ceph-users] RGW listing users' quota and usage painfully slow

2017-03-09 Thread Matthew Vernon
On 09/03/17 11:28, Matthew Vernon wrote: https://drive.google.com/drive/folders/0B4TV1iNptBAdMEdUaGJIa3U1QVE?usp=sharing [For the avoidance of doubt, I've changed the key associated with that S3 account :-) ] Regards, Matthew -- The Wellcome Trust Sanger Institute is operated by Genome R

Re: [ceph-users] [Jewel] upgrade 10.2.3 => 10.2.5 KO : first OSD server freeze every two days :)

2017-03-09 Thread Vincent Godin
First of all, don't do a ceph upgrade while your cluster is in warning or error state. A process upgrade must be done from an clean cluster. Don't stay with a replicate at 2. Majority of problems come from that point: just look the advices given by experience users of the list. You should set a re

Re: [ceph-users] RGW listing users' quota and usage painfully slow

2017-03-09 Thread Orit Wasserman
On Thu, Mar 9, 2017 at 1:28 PM, Matthew Vernon wrote: > On 09/03/17 10:45, Abhishek Lekshmanan wrote: > > On 03/09/2017 11:26 AM, Matthew Vernon wrote: >> >>> >>> I'm using Jewel / 10.2.3-0ubuntu0.16.04.2 . We want to keep track of our >>> S3 users' quota and usage. Even with a relatively small n

Re: [ceph-users] [Jewel] upgrade 10.2.3 => 10.2.5 KO : first OSD server freeze every two days :)

2017-03-09 Thread pascal.pu...@pci-conseil.net
Le 09/03/2017 à 13:03, Vincent Godin a écrit : First of all, don't do a ceph upgrade while your cluster is in warning or error state. A process upgrade must be done from an clean cluster. of course. So, Yesterday, so I try this for my "unfound PG" ceph pg 50.2dd mark_unfound_lost revert => MON

[ceph-users] pgs stuck inactive

2017-03-09 Thread Laszlo Budai
Hello, After a major network outage our ceph cluster ended up with an inactive PG: # ceph health detail HEALTH_WARN 1 pgs incomplete; 1 pgs stuck inactive; 1 pgs stuck unclean; 1 requests are blocked > 32 sec; 1 osds have slow requests pg 3.367 is stuck inactive for 912263.766607, current state

Re: [ceph-users] cephfs and erasure coding

2017-03-09 Thread Rhian Resnick
Thanks for the confirmations of what is possible. We plan on creating a new file system, rsync and delete the old one. Rhian On Mar 9, 2017 2:27 AM, Maxime Guyot wrote: Hi, >“The answer as to how to move an existing cephfs pool from replication to >erasure coding (and vice versa) is to cre

Re: [ceph-users] How does ceph preserve read/write consistency?

2017-03-09 Thread Wei Jin
On Thu, Mar 9, 2017 at 1:45 PM, 许雪寒 wrote: > Hi, everyone. > As shown above, WRITE req with tid 1312595 arrived at 18:58:27.439107 and > READ req with tid 6476 arrived at 18:59:55.030936, however, the latter > finished at 19:00:20:89 while the former finished commit at > 19:00:20.335061 an

Re: [ceph-users] Object Map Costs (Was: Snapshot Costs (Was: Re: Pool Sizes))

2017-03-09 Thread Kent Borg
On 03/08/2017 05:07 PM, Gregory Farnum wrote: How about iterating through a whole set of values vs. reading a RADOS object holding the same amount of data? "Iterating"? As in rados_read_op_omap_get_vals(), "Start iterating over key/value pairs on an object." In general, you should use the

Re: [ceph-users] Bogus "inactive" errors during OSD restarts with Jewel

2017-03-09 Thread Ruben Kerkhof
On Thu, Mar 9, 2017 at 3:04 AM, Christian Balzer wrote: > > > Hello, > > during OSD restarts with Jewel (10.2.5 and .6 at least) I've seen > "stuck inactive for more than 300 seconds" errors like this when observing > things with "watch ceph -s" : > --- > health HEALTH_ERR > 59 pg

[ceph-users] Ceph with RDMA

2017-03-09 Thread PR PR
Hi, I am trying to use ceph with RDMA. I have a few questions. 1. Is there a prebuilt package that has rdma support or the only way to try ceph+rdma is to checkout from github and compile from scratch? 2. Looks like there are two ways of using rdma - xio and async+rdma. Which is the recommended

Re: [ceph-users] cephfs and erasure coding

2017-03-09 Thread Rhian Resnick
Thanks everyone for the input. We are online in our test environment and are running user workflows to make sure everything is running as expected. Rhian From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Rhian Resnick Sent: Thursday, March 9, 2017 8:31 AM To: Maxime Guy

Re: [ceph-users] Shrinking lab cluster to free hardware for a new deployment

2017-03-09 Thread Ben Hines
AFAIK depending on how many you have, you are likely to end up with 'too many pgs per OSD' warning for your main pool if you do this, because the number of PGs in a pool cannot be reduced and there will be less OSDs to put them on. -Ben On Wed, Mar 8, 2017 at 5:53 AM, Henrik Korkuc wrote: > On

Re: [ceph-users] Object Map Costs (Was: Snapshot Costs (Was: Re: Pool Sizes))

2017-03-09 Thread Max Yehorov
re: python library you can do some mon calls using this: ##-- from ceph_argparse import json_command as json_command rados_inst = rados.Rados() cluster_handle = rados_inst.connect() cmd = {'prefix': 'pg dump', 'dumpcontents': ['summary', ], 'format': 'json'} retcode, jsonret, errstr

Re: [ceph-users] Why is librados for Python so Neglected?

2017-03-09 Thread Max Yehorov
There was definitely missing the "list watchers" command. On Wed, Mar 8, 2017 at 4:16 PM, Josh Durgin wrote: > On 03/08/2017 02:15 PM, Kent Borg wrote: >> >> On 03/08/2017 05:08 PM, John Spray wrote: >>> >>> Specifically? >>> I'm not saying you're wrong, but I am curious which bits in particular

Re: [ceph-users] Why is librados for Python so Neglected?

2017-03-09 Thread Kent Borg
On 03/09/2017 06:19 PM, Max Yehorov wrote: There was definitely missing the "list watchers" command. I was hearing something about how somebody doing locking partly via list watchers...but hadn't looked for it. The Python librados I have been playing with was part of Debian. Today I built i

Re: [ceph-users] pgs stuck inactive

2017-03-09 Thread Brad Hubbard
Can you explain more about what happened? The query shows progress is blocked by the following OSDs. "blocked_by": [ 14, 17, 51, 58, 63, 64,

Re: [ceph-users] Ceph with RDMA

2017-03-09 Thread Haomai Wang
On Fri, Mar 10, 2017 at 4:28 AM, PR PR wrote: > Hi, > > I am trying to use ceph with RDMA. I have a few questions. > > 1. Is there a prebuilt package that has rdma support or the only way to try > ceph+rdma is to checkout from github and compile from scratch? > > 2. Looks like there are two ways o

[ceph-users] 答复: How does ceph preserve read/write consistency?

2017-03-09 Thread 许雪寒
Thanks for your reply. As the log shows, in our test, a READ that come after a WRITE did finished before that WRITE. And I read the source code, it seems that, for writes, in ReplicatedPG::do_op method, the thread in OSD_op_tp calls ReplicatedPG::get_rw_lock method which tries to get RWState::R

[ceph-users] 答复: 答复: How does ceph preserve read/write consistency?

2017-03-09 Thread 许雪寒
I also submitted an issue: http://tracker.ceph.com/issues/19252 -邮件原件- 发件人: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] 代表 许雪寒 发送时间: 2017年3月10日 11:20 收件人: Wei Jin; ceph-users@lists.ceph.com 主题: [ceph-users] 答复: How does ceph preserve read/write consistency? Thanks for your reply

Re: [ceph-users] Posix AIO vs libaio read performance

2017-03-09 Thread Xavier Trilla
Hi, We compiled Hammer .10 to use jemalloc and now the cluster performance improved a lot, but POSIX AIO operations are still quite slower than libaio. Now with a single thread read operations are about 1000 per second and write operations about 5000 per second. Using same FIO configuration,

Re: [ceph-users] Posix AIO vs libaio read performance

2017-03-09 Thread Alexandre DERUMIER
>>But it still looks like there is some bottleneck in QEMU o Librbd I cannot >>manage to find. you can improve latency on client with disable debug. on your client, create a /etc/ceph/ceph.conf with [global] debug asok = 0/0 debug auth = 0/0 debug buffer = 0/0 debug client = 0/0 debug con