Hello, Adam. Try to attach md raid0 of two Ceph vols inside VM. I have heard that people who run ceph use that practice to improve results. I don't know what you are expecting, but read somewhere that all ops to a single RBD volume are queued in a single IO queue, thus RAID0 helps here, increasing the speed almost twice. I mean you don't see a ceph cluster overall performance, you see a librbd+qemu performance in your tests.
вт, 26 июн. 2018 г., 15:57 Adam Witwicki <[email protected]>: > We are days away from completing our production CloudStack / Ceph > deployment but have run into a small challenge in regards to Ceph > performance. As you run a setup similar to ours I was wondering if you’ve > seen a similar issue? > > > > Our Ceph cluster consists of 60 X 7.2K SAS drive, spread over 5 nodes each > with a 960GB Samsung NVMe. Our KVM hypervisors are connected to this nodes > via 2 x 10Gbps links, per node. > > > > 3 centos Cloudstack management servers 4.11.0 > > 4 centos KVM hosts > > > > The RAW Ceph performance looks great! > > > > CEPH RAW > > *Average IOPS* > > *Max IOPS* > > *MB/sec* > > *Write 4K* > > 5,885 > > 6,190 > > 22.9907 > > *Read 4K* > > 28,985 > > 35,025 > > 113 > > *Write 4MB* > > 204 > > 219 > > 816 > > *Read 4MB* > > 361 > > 399 > > 1447.51 > > > > However when we run the tests from within KVM we get significantly reduced > performance, particular in writes: > > > > VM Running on KVM Hypervisor > > *Average IOPS* > > *Max IOPS* > > *MB/sec* > > *Write 4K* > > 946.07 > > 1390 > > 3.69 > > *Write 4MB* > > 472 > > 614 > > 116 > > > > We believe the libvirt/ceph library could be causing the problem with in > centos as we don’t see this on a Ubuntu KVM HOST > > > > Kind Regards, > > [image: Oakford Internet Services] <http://www.oakfordis.com/> > > *Adam Witwicki* | Hosted Systems Specialist > > 01380 710278 / [email protected] > > *Oakford Internet Services* Office: 01380 888088 > 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT > www.oakfordis.com [email protected] > > > > Disclaimer Notice: > This email has been sent by Oakford Technology Limited, while we have > checked this e-mail and any attachments for viruses, we can not guarantee > that they are virus-free. You must therefore take full responsibility for > virus checking. > This message and any attachments are confidential and should only be read > by those to whom they are addressed. If you are not the intended recipient, > please contact us, delete the message from your computer and destroy any > copies. Any distribution or copying without our prior permission is > prohibited. > Internet communications are not always secure and therefore Oakford > Technology Limited does not accept legal responsibility for this message. > The recipient is responsible for verifying its authenticity before acting > on the contents. Any views or opinions presented are solely those of the > author and do not necessarily represent those of Oakford Technology Limited. > Registered address: Oakford Technology Limited, 10 Prince Maurice Court, > Devizes, Wiltshire. SN10 2RT. > Registered in England and Wales No. 5971519 >
