I have a newly setup test cluster that is giving some surprising numbers when
running fio against an RBD. The end goal here is to see how viable a Ceph based
iSCSI SAN of sorts is for VMware clusters, which require a bunch of random IO.
Hardware:
2x E5-2630L v2 (2.4GHz, 6 core)
256GB RAM
2x 10gbps bonded network, Intel X520
LSI 9271-8i, SSDs used for OSDs in JBOD mode
Mons: 2x 1.2TB 10K SAS in RAID1
OSDs: 12x Samsung MZ6ER800HAGL-00003 800GB SAS SSDs, super cap/power loss
protection
Cluster setup:
Three mon nodes, four OSD nodes
Two OSDs per SSD
Replica 3 pool
Ceph 14.2.5
Ceph status:
cluster:
id: e3d93b4a-520c-4d82-a135-97d0bda3e69d
health: HEALTH_WARN
application not enabled on 1 pool(s)
services:
mon: 3 daemons, quorum mon1,mon2,mon3 (age 6d)
mgr: mon2(active, since 6d), standbys: mon3, mon1
osd: 96 osds: 96 up (since 3d), 96 in (since 3d)
data:
pools: 1 pools, 3072 pgs
objects: 857.00k objects, 1.8 TiB
usage: 432 GiB used, 34 TiB / 35 TiB avail
pgs: 3072 active+clean
Network between nodes tests at 9.88gbps. Direct testing of the SSDs using a 4K
block in fio shows 127k seq read, 86k randm read, 107k seq write, 52k random
write. No high CPU load/interface saturation is noted when running tests
against the rbd.
When testing with a 4K block size against an RBD on a dedicated metal test host
(same specs as other cluster nodes noted above) I get the following (command
similar to fio -ioengine=rbd -direct=1 -name=test -bs=4k -iodepth=32 -rw=XXXX
-pool=scbench -runtime=60 -rbdname=datatest):
10k sequential read iops
69k random read iops
13k sequential write iops
22k random write iops
I’m not clear why the random ops, especially read, would be so much quicker
compared to the sequential ops.
Any points appreciated.
Thanks,
Anthony
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
