Re: [ceph-users] osd_op_tp timeouts

2017-06-13 Thread Eric Choi
I realized I sent this under wrong thread: here I am sending it again: --- Hello all, I work in the same team as Tyler here, and I can provide more info here.. The cluster is indeed an RGW cluster, with many small (100 KB) objects similar to your use case Bryan. But we have the blind bucket se

Re: [ceph-users] osd_op_tp timeouts

2017-06-13 Thread Bryan Stillwell
Is this on an RGW cluster? If so, you might be running into the same problem I was seeing with large bucket sizes: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-June/018504.html The solution is to shard your buckets so the bucket index doesn't get too big. Bryan From: ceph-users o

Re: [ceph-users] osd_op_tp timeouts

2017-06-13 Thread Mark Nelson
Hi Tyler, I wanted to make sure you got a reply to this, but unfortunately I don't have much to give you. It sounds like you already took a look at the disk metrics and ceph is probably not waiting on disk IO based on your description. If you can easily invoke the problem, you could attach g