Interesting. We've seen things like this on the librbd side in the
past, but I don't think I've seen this kind of behavior in the kernel
client. what does the latency historgram look like when going from 1->2?
Mark
On 03/06/2015 08:10 AM, Nick Fisk wrote:
Just tried cfq, deadline and noop which more or less all show identical results
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Alexandre DERUMIER
Sent: 06 March 2015 11:59
To: Nick Fisk
Cc: ceph-users
Subject: Re: [ceph-users] Strange krbd behaviour with queue depths
Hi, do you have tried with differents io schedulers to compare ?
----- Mail original -----
De: "Nick Fisk" <n...@fisk.me.uk>
À: "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Jeudi 5 Mars 2015 18:17:27
Objet: [ceph-users] Strange krbd behaviour with queue depths
I’m seeing a strange queue depth behaviour with a kernel mapped RBD, librbd
does not show this problem.
Cluster is comprised of 4 nodes, 10GB networking, not including OSDs as test
sample is small so fits in page cache.
Running fio against a kernel mapped RBD
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=/dev/rbd/cache1/test2 --bs=4k --readwrite=randread --iodepth=1
--runtime=10 --size=1g
Queue Depth
IOPS
1
2021
2
288
4
376
8
601
16
1272
32
2467
64
16901
128
44060
See how initially I get a very high number of IOs at queue depth 1, but this
drops dramatically as soon as I start increasing the queue depth. It’s not
until a depth or 32 IOs that I start to get similar performance. Incidentally
when changing the read type to sequential instead of random the oddity goes
away.
Running fio with librbd engine and the same test options I get the following
Queue Depth
IOPS
1
1492
2
3232
4
7099
8
13875
16
18759
32
17998
64
18104
128
18589
As you can see the performance scales up nicely, although the top end IO’s seem
limited to around 18k. I don’t know if this is due to kernel/userspace
performance differences or if there is a lower max queue depth limit in librbd.
Both tests were run on a small sample size to force the OSD data into page
cache to rule out any device latency.
Does anyone know why kernel mapped RBD’s show this weird behaviour? I don’t
think it can be OSD/ceph config related as it only happens with krbd’s.
Nick
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Nick Fisk
Technical Support Engineer
System Professional Ltd
tel: 01825 830000
mob: 07711377522
fax: 01825 830001
mail: nick.f...@sys-pro.co.uk
web: www.sys-pro.co.uk<http://www.sys-pro.co.uk>
IT SUPPORT SERVICES | VIRTUALISATION | STORAGE | BACKUP AND DR | IT CONSULTING
Registered Office:
Wilderness Barns, Wilderness Lane, Hadlow Down, East Sussex, TN22 4HU
Registered in England and Wales.
Company Number: 04754200
Confidentiality: This e-mail and its attachments are intended for the above
named only and may be confidential. If they have come to you in error you must
take no action based on them, nor must you copy or show them to anyone; please
reply to this e-mail and highlight the error.
Security Warning: Please note that this e-mail has been created in the
knowledge that Internet e-mail is not a 100% secure communications medium. We
advise that you understand and observe this lack of security when e-mailing us.
Viruses: Although we have taken steps to ensure that this e-mail and
attachments are free from any virus, we advise that in keeping with good
computing practice the recipient should ensure they are actually virus free.
Any views expressed in this e-mail message are those of the individual and not
necessarily those of the company or any of its subsidiaries.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
