Hello, harddisks are awful slow. That's normal and expected and result of random io as it would be in a Ceph cluster. You can speed up raw bandwidth performance using EC but not on such small clusters and not when having high io load.
As you mentioned Proxmox, when it comes to VM workloads spinning media is never an option, use Flash! -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.ver...@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Mo., 4. Nov. 2019 um 23:44 Uhr schrieb Hermann Himmelbauer < herm...@qwer.tk>: > Hi, > I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and > recreated my ceph cluster with a new release (14.2.4 bluestore) - > basically hoping to gain some I/O speed. > > The installation went flawlessly, reading is faster than before (~ 80 > MB/s), however, the write speed is still really slow (~ 3,5 MB/s). > > I wonder if I can do anything to speed things up? > > My Hardware is as the following: > > 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, > 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), > interconnected via Infiniband 40 > > The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, > so this seems fine. > > I use ceph for RBD only, so my measurement is simply doing a very simple > "dd" read and write test within a virtual machine (Debian 8) like the > following: > > read: > dd if=/dev/vdb | pv | dd of=/dev/null > -> 80 MB/s > > > write: > dd if=/dev/zero | pv | dd of=/dev/vdb > -> 3.5 MB/s > > When I do the same on the virtual machine on a disk that is on a NFS > storage, I get something about 30 MB/s for reading and writing. > > If I disable the write cache on all OSD disks via "hdparm -W 0 > /dev/sdX", I gain a little bit of performance, write speed is then 4.3 > MB/s. > > Thanks to your help from the list I plan to install a second ceph > cluster which is SSD based (Samsung PM1725b) which should be much > faster, however, I still wonder if there is any way to speed up my > harddisk based cluster? > > Thank you in advance for any help, > > Best Regards, > Hermann > > > -- > herm...@qwer.tk > PGP/GPG: 299893C7 (on keyservers) > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io >
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
