On 01/24/2014 09:31 AM, Christian Kauhaus wrote:
Hi,
we're using Ceph to serve VM images via RBD and thus, RBD performance is
important for us. I've prepared some write benchmarks using different object
sizes. One time I use 'rados bench' directly and the other time 'rbd
bench-write'.
The results are interesting: Raw RADOS write rates are significantly better
for large objects (>128k), RBD performs better for medium sized objects (>16k,
<128k), but RBD is really slow for small writes. We have lot of small writes,
so this is the pain point. I think latencies are dominant here.
Our test setup consists of two Ceph servers running a MON and 9 OSDs (one OSD
daemon per disk; ext4 filesystem) with journals on a shared SSD (one SSD
partition per OSD). There are 2 GigE networks (storage frontend/backend) with
approx 62µs RTT and jumbo frames enabled. See attached ceph.conf further
details. Some parameters there are taken from the tuning recommendations at
[1]. Note that I have to stick to ext4 on the OSDs.
Is there anything we can do to improve latencies? I don't know where to start:
* OSD setup?
* Network setup?
* ceph.conf parameter tuning?
* Separate MONs?
* Separate networks for MON access?
A lot of options... so I would be grateful for hints what is worth looking at.
Are you using RBD cache? That can help quite a bit for small writes.
More generally: How many IOPs per disk are you getting now and what are
your goals? Like you said, there are many many different things to look
at, and IOPS is often trickier to tune for.
Please refer to bitbucket[2] for benchmark scripts.
TIA
Christian
[1] http://ceph.com/community/ceph-bobtail-jbod-performance-tuning/
[2] https://bitbucket.org/ckauhaus/ceph_performance
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
