I recently created a 9-node Firefly cluster backed by all SSDs. We have had some pretty severe performance degradation when using O_DIRECT in our tests (as this is how MySQL will be interacting with RBD volumes, this makes the most sense for a preliminary test). Running the following test:
dd if=/dev/zero of=testfilasde bs=16k count=65535 oflag=direct 779829248 bytes (780 MB) copied, 604.333 s, 1.3 MB/s Shows us only about 1.5 MB/s throughput and 100 IOPS from the single dd thread. Running a second dd process does show increased throughput which is encouraging, but I am still concerned by the low throughput of a single thread w/ O_DIRECT. Two threads: 779829248 bytes (780 MB) copied, 604.333 s, 1.3 MB/s 126271488 bytes (126 MB) copied, 99.2069 s, 1.3 MB/s I am testing with an RBD volume mounted with the kernel module (I have also tested from within KVM, similar performance). If allow caching, we start to see reasonable numbers from a single dd process: dd if=/dev/zero of=testfilasde bs=16k count=65535 65535+0 records in 65535+0 records out 1073725440 bytes (1.1 GB) copied, 2.05356 s, 523 MB/s I can get >1GB/s from a single host with three threads. Rados bench produces similar results. Is there something I can do to increase the performance of O_DIRECT? I expect performance degradation, but so much? If I increase the blocksize to 4M, I'm able to get significantly higher throughput: 3833593856 bytes (3.8 GB) copied, 44.2964 s, 86.5 MB/s This still seems very low. I'm using the deadline scheduler in all places. With noop scheduler, I do not see a performance improvement. Suggestions?
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
