There have been some bugs in the past with dynamic resharding, but those seem to be the exception not the norm. I was surprised to find that some of our buckets had resharded themselves from 12 shards in Jewel to over 200 shards in luminous without me even realizing it. We would have resharded in Jewel, but were on a version where it was bugged and it wasn't possible without upgrading. When I went back to reshard later after the upgrade to luminous I found that the buckets had all resharded themselves without any problems. When we have blocked requests on our nvme osds (where the RGW metadata lives) I always check if any resharding is happening, but it never is. I haven't tracked any problems down to resharding.
On Tue, Jun 19, 2018 at 3:44 PM Matt Benjamin <mbenj...@redhat.com> wrote: > The increased time to list sharded buckets is currently expected, yes. > In turn other operations such as put and delete should be faster in > proportion to two factors, the number of shards on independent PGs > (serialization by PG), and the spread of shards onto independent OSD > devices (speedup from scaling onto more OSD devices, presuming > available iops on those devices). > > New bucket index formats are coming in future to help listing > workloads. Also, as of recent master (and probably Jewel and Luminous > at this point, modulo some latency for the backports) we have added an > "allow-unordered" option to S3 and Swift listing arguments that should > remove the penalty from sharding. This causes results to be returned > in partial order, rather than the total order most applications > expect. > > Matt > > On Tue, Jun 19, 2018 at 9:34 AM, Matthew Vernon <m...@sanger.ac.uk> wrote: > > Hi, > > > > Some of our users have Quite Large buckets (up to 20M objects in a > > bucket), and AIUI best practice would be to have sharded indexes for > > those buckets (of the order of 1 shard per 100k objects). > > > > On a trivial test case (make a 1M-object bucket, shard index to 10 > > shards, s3cmd ls s3://bucket >/dev/null), sharding makes the bucket > > listing slower (not a lot, but a bit). > > > > Are there simple(ish) workflows we could use to demonstrate an > > improvement from index sharding? > > > > Thanks, > > > > Matthew > > > > [I understand that Luminous has dynamic resharding, but it seems a bit > > unstable for production use; is that still the case?] > > > > > > -- > > The Wellcome Sanger Institute is operated by Genome Research > > Limited, a charity registered in England with number 1021457 and a > > company registered in England with number 2742969, whose registered > > office is 215 Euston Road, London, NW1 2BE. > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 <(734)%20821-5101> > fax. 734-769-8938 <(734)%20769-8938> > cel. 734-216-5309 <(734)%20216-5309> > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
