> I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is > having low IO whereas it’s also on all NVME. > What to tune to reach equal amount of IO? > > root@node01:~/fio-cdm# python3 fio-cdm ./ > tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB
You should really make sure the size of the disk object you make tests on are larger than available ram. I don't know what options your fio-cdm script gives, but 1G is super easy for a machine to cache in ram, so that could skew the tests totally. Apart from that, it will ALWAYS be slower to read data across network (even if network to localhost), and if the RBD is on a replicated pool, then of course it will need to wait for acks on the replicas before allowing the client to know the write has been done, which again slows it down more. Finally, the SEQ1Q tests sounds a lot like as if it was like running "dd" against ceph which is basically the worst possible kind of test for a cluster network storage system. Firstly, it is basically something very very few clients need optimizing for, linear dd with no queuing means "write 1 meg, wait for ack, do nothing until ack comes, then write 1 meg more and go back to waiting". This says more or less nothing about a storage cluster which ties in to the second part, what to use a cluster for. Most clusters are used to handle many clients doing cephfs, or many clients each mounting one or more RBD images, or many RGW frontends each serving many parallel client sessions. So even if each other those clients would be doing the linear "dance" of "write, wait, wait, write, wait, wait, write", the cluster in total would obviously have a much larger throughput, because some other client will of course be reading and writing to other disks while first client is waiting for acks. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
