On 01/10/2014 03:08 AM, Bradley Kite wrote:
On 9 January 2014 16:57, Mark Nelson <mark.nel...@inktank.com
<mailto:mark.nel...@inktank.com>> wrote:
On 01/09/2014 10:43 AM, Bradley Kite wrote:
On 9 January 2014 15:44, Christian Kauhaus <k...@gocept.com
<mailto:k...@gocept.com>
<mailto:k...@gocept.com <mailto:k...@gocept.com>>> wrote:
Am 09.01.2014 10:25, schrieb Bradley Kite:
> 3 servers (quad-core CPU, 16GB RAM), each with 4 SATA
7.2K RPM
disks (4TB)
> plus a 160GB SSD.
> [...]
> By comparison, a 12-disk RAID5 iscsi SAN is doing ~4000
read iops
and ~2000
> iops write (but with 15KRPM SAS disks).
I think that comparing Ceph on 7.2k rpm SATA disks against
iSCSI on
15k rpm
SAS disks is not fair. The random access times of 15k SAS
disks are
hugely
better compared to 7.2k SATA disks. What would be far more
interesting is to
compare Ceph against iSCSI with identical disks.
Regards
Christian
--
Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com
<mailto:k...@gocept.com>
<mailto:k...@gocept.com <mailto:k...@gocept.com>> · systems
administration
gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle
(Saale) · Germany
http://gocept.com · tel +49 345 219401-11
<tel:%2B49%20345%20219401-11> <tel:%2B49%20345%20219401-11>
Python, Pyramid, Plone, Zope · consulting, development,
hosting,
operations
_________________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
<mailto:ceph-us...@lists.ceph.__com
<mailto:ceph-users@lists.ceph.com>>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
Hi Christian,
Yes, for a true comparison it would be better but this is the
only iscsi
SAN that we have available for testing, so I really only compared
against it to get a "gut feel" for relative performance.
I'm still looking for clues that might indicate why there is
such a huge
difference between the read & write rates on the ceph cluster
though.
One thing you may want to look at is some comparisons we did with
fio on different RBD volumes with varying io depths and volume/guest
counts:
http://ceph.com/performance-2/__ceph-cuttlefish-vs-bobtail-__part-2-4k-rbd-performance/
<http://ceph.com/performance-2/ceph-cuttlefish-vs-bobtail-part-2-4k-rbd-performance/>
You'll probably be most interested in the 4k random read/write
results for XFS. It would be interesting to see if you saw any
difference with more or less volumes at different io depths. Also,
sorry if I missed it, but is this QEMU/KVM? If so, did you enable
RBD cache?
Hi Mark,
Thanks for your very detailed test results.
Your results are interesting, and suggest that there is a significant
performance difference between kernel RBD mapping and QEMU/KVM (which
uses librbd directly) - shown particularly here where KRBD achieves
23MB/sec vs librbd 500 MB/sec:
http://ceph.com/wp-content/uploads/2014/07/cuttlefish-rbd_xfs-write-0004K.png
Particularly for small sequential IO with the direct flag, RBD cache
does help dramatically as it allow all of those little sequential IOs to
be coalesced. I suspect you will notice some benefit with random direct
IO (assuming you are using moderate sized VM images), but not nearly as
much. Buffered IO is interesting because now you have the linux buffer
cache involved and the results may be quite different.
Our end-goal is to use QEMU/KVM so this is very promising. Would you
happen to have the raw iops/second figures from your tests? The graphs
only show throughput which provides a good comparison but for us IOPS is
the most important factor.
I'll have to check and see if I can get you the original data. You can
manually figure out the iops too by just taking the MB/s throughput and
the io size and dividing: X MB/s * 1024 / 4KB = Y IOPS (4KB)
Would you happen to know if stgt (iscsi) uses the kernel module or
librbd? We also have some legacy HyperV hosts that we would like to
connect (to avoid rebuilding them).
Not sure. I suspect it's not using any of the kernel code, but someone
else can probably say for sure.
Is it generally recommended to avoid the kernel module where possible?
Not necessarily, but there are trade-offs. The kernel module may be
faster in some ways as it's more bare-metal, but it's also much more
difficult to implement things like write back caching and other features
in it. I would stick with the QEMU driver if you can use it, but the
kernel module may have advantages in some situations (say if you do
entirely sequential large reads/writes).
Regards
--
Brad.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com