20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming replica 3).
What we usually check in scenarios like these: * SSD model? Lots of cheap SSDs simply can't handle more than that * Get some proper statistics such as OSD latencies, disk IO utilization, etc. A benchmark without detailed performance data doesn't really help to debug such a problem Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, May 24, 2019 at 1:23 PM Robert Sander <r.san...@heinlein-support.de> wrote: > Hi, > > we have a small cluster at a customer's site with three nodes and 4 > SSD-OSDs each. > Connected with 10G the system is supposed to perform well. > > rados bench shows ~450MB/s write and ~950MB/s read speeds with 4MB > objects but only 20MB/s write and 95MB/s read with 4KB objects. > > This is a little bit disappointing as the 4K performance is also seen in > KVM VMs using RBD. > > Is there anything we can do to improve performance with small objects / > block sizes? > > Jumbo frames have already been enabled. > > 4MB objects write: > > Total time run: 30.218930 > Total writes made: 3391 > Write size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 448.858 > Stddev Bandwidth: 63.5044 > Max bandwidth (MB/sec): 552 > Min bandwidth (MB/sec): 320 > Average IOPS: 112 > Stddev IOPS: 15 > Max IOPS: 138 > Min IOPS: 80 > Average Latency(s): 0.142475 > Stddev Latency(s): 0.0990132 > Max latency(s): 0.814715 > Min latency(s): 0.0308732 > > 4MB objects rand read: > > Total time run: 30.169312 > Total reads made: 7223 > Read size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 957.662 > Average IOPS: 239 > Stddev IOPS: 23 > Max IOPS: 272 > Min IOPS: 175 > Average Latency(s): 0.0653696 > Max latency(s): 0.517275 > Min latency(s): 0.00201978 > > 4K objects write: > > Total time run: 30.002628 > Total writes made: 165404 > Write size: 4096 > Object size: 4096 > Bandwidth (MB/sec): 21.5351 > Stddev Bandwidth: 2.0575 > Max bandwidth (MB/sec): 22.4727 > Min bandwidth (MB/sec): 11.0508 > Average IOPS: 5512 > Stddev IOPS: 526 > Max IOPS: 5753 > Min IOPS: 2829 > Average Latency(s): 0.00290095 > Stddev Latency(s): 0.0015036 > Max latency(s): 0.0778454 > Min latency(s): 0.00174262 > > 4K objects read: > > Total time run: 30.000538 > Total reads made: 1064610 > Read size: 4096 > Object size: 4096 > Bandwidth (MB/sec): 138.619 > Average IOPS: 35486 > Stddev IOPS: 3776 > Max IOPS: 42208 > Min IOPS: 26264 > Average Latency(s): 0.000443905 > Max latency(s): 0.0123462 > Min latency(s): 0.000123081 > > > Regards > -- > Robert Sander > Heinlein Support GmbH > Linux: Akademie - Support - Hosting > http://www.heinlein-support.de > > Tel: 030-405051-43 > Fax: 030-405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
