On 05/26/2015 08:53 AM, lixuehui...@126.com wrote:
Hi ALL:
Hi!
I've built a ceph0.8 cluster including 2 nodes ,which contains 5
osds(ssd) each , with 100MB/s network . Testing a rbd device with
default configuration ,the result is no ideal.To got better performance
,except the capability of random r/w of SSD, which should to give a
change?
2 nodes 5 osds(SSD) *2 , 1 mon, 32GB RAM
100MB/S network
and now the whole iops is just 500 . Should we change the filestore or
journal part ? Thanks for any help!
------------------------------------------------------------------------
lixuehui...@126.com
For writes, it's important that the SSD be able to process O_DSYNC
writes quickly if it's being used for journal writes. The best ones
have power-loss-protection that allows them to ignore flush requests and
continue writing without pause. Be careful though because not all SSDs
do this properly.
You can attempt to simulate how the OSD journal performs writes using
fio. Keep in mind that each write has a small amount of header
information and is padded out to the nearest 4K boundary. For 4K writes
on SSDs for instance, you may be important. Keeping that in mind, in
the 4K case, something like this might be a good approximation:
fio --rw=write --ioengine=libaio --numjobs=1 --direct=1 --runtime=300
--bssplit=84k/20:88k/20:92k/20:96k/20:100k/20 --sync=1
--iodepth=9*$journals_per_device [ --name $device_name ]
Beyond that, We also spent some time looking at performance in the
hammer development process on a couple of different SSDs:
http://nhm.ceph.com/Ceph_SSD_OSD_Performance.pdf
http://nhm.ceph.com/Ceph_Hammer_OSD_Shard_Tuning_PCIe.pdf
Those are just single-OSD tests. Once the network is involved things
can get more complicated. Still, there's some useful data there. Look
in the appendices for the configuration used during testing. The basic
gist of it is that these things may help:
1) Disabling in-memory logging
2) Disabling authentication
3) Tweaking the OSD sharding (Likely most relevant for very fast SSDs)
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
