Greg, Not a real fix for you but I too run a full-ssd cluster and am able to get 112MB/s with your command;
[root@plesk-test ~]# dd if=/dev/zero of=testfilasde bs=16k count=65535 oflag=direct 65535+0 records in 65535+0 records out 1073725440 bytes (1.1 GB) copied, 9.59092 s, 112 MB/s This of course is in a VM, here is my ceph config [global] fsid = <hidden> mon_initial_members = node-1 node-2 node-3 mon_host = 192.168.0.3 192.168.0.4 192.168.0.5 auth_supported = cephx osd_journal_size = 2048 filestore_xattr_use_omap = true osd_pool_default_size = 2 osd_pool_default_min_size = 1 osd_pool_default_pg_num = 1024 public_network = 192.168.0.0/24 osd_mkfs_type = xfs cluster_network = 192.168.1.0/24 On Fri, Jun 20, 2014 at 11:08 AM, Greg Poirier <greg.poir...@opower.com> wrote: > I recently created a 9-node Firefly cluster backed by all SSDs. We have > had some pretty severe performance degradation when using O_DIRECT in our > tests (as this is how MySQL will be interacting with RBD volumes, this > makes the most sense for a preliminary test). Running the following test: > > dd if=/dev/zero of=testfilasde bs=16k count=65535 oflag=direct > > 779829248 bytes (780 MB) copied, 604.333 s, 1.3 MB/s > > Shows us only about 1.5 MB/s throughput and 100 IOPS from the single dd > thread. Running a second dd process does show increased throughput which is > encouraging, but I am still concerned by the low throughput of a single > thread w/ O_DIRECT. > > Two threads: > 779829248 bytes (780 MB) copied, 604.333 s, 1.3 MB/s > 126271488 bytes (126 MB) copied, 99.2069 s, 1.3 MB/s > > I am testing with an RBD volume mounted with the kernel module (I have > also tested from within KVM, similar performance). > > If allow caching, we start to see reasonable numbers from a single dd > process: > > dd if=/dev/zero of=testfilasde bs=16k count=65535 > 65535+0 records in > 65535+0 records out > 1073725440 bytes (1.1 GB) copied, 2.05356 s, 523 MB/s > > I can get >1GB/s from a single host with three threads. > > Rados bench produces similar results. > > Is there something I can do to increase the performance of O_DIRECT? I > expect performance degradation, but so much? > > If I increase the blocksize to 4M, I'm able to get significantly higher > throughput: > > 3833593856 bytes (3.8 GB) copied, 44.2964 s, 86.5 MB/s > > This still seems very low. > > I'm using the deadline scheduler in all places. With noop scheduler, I do > not see a performance improvement. > > Suggestions? > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
