We actually do have a use pattern of large batch sequential writes, and this dd is pretty similar to that use case.
A round-trip write with replication takes approximately 10-15ms to complete. I've been looking at dump_historic_ops on a number of OSDs and getting mean, min, and max for sub_op and ops. If these were on the order of 1-2 seconds, I could understand this throughput... But we're talking about fairly fast SSDs and a 20Gbps network with <1ms latency for TCP round-trip between the client machine and all of the OSD hosts. I've gone so far as disabling replication entirely (which had almost no impact) and putting journals on separate SSDs as the data disks (which are ALSO SSDs). This just doesn't make sense to me. On Sun, Jun 22, 2014 at 6:44 AM, Mark Nelson <mark.nel...@inktank.com> wrote: > On 06/22/2014 02:02 AM, Haomai Wang wrote: > >> Hi Mark, >> >> Do you enable rbdcache? I test on my ssd cluster(only one ssd), it seemed >> ok. >> >> dd if=/dev/zero of=test bs=16k count=65536 oflag=direct >>> >> >> 82.3MB/s >> > > RBD Cache is definitely going to help in this use case. This test is > basically just sequentially writing a single 16k chunk of data out, one at > a time. IE, entirely latency bound. At least on OSDs backed by XFS, you > have to wait for that data to hit the journals of every OSD associated with > the object before the acknowledgement gets sent back to the client. If you > are using the default 4MB block size, you'll hit the same OSDs over and > over again and your other OSDs will sit there twiddling their thumbs > waiting for IO until you hit the next block, but then it will just be a > different set OSDs getting hit. You should be able to verify this by using > iostat or collectl or something to look at the behaviour of the SSDs during > the test. Since this is all sequential though, switching to buffered IO > (ie coalesce IOs at the buffercache layer) or using RBD cache for direct IO > (coalesce IOs below the block device) will dramatically improve things. > > The real question here though, is whether or not a synchronous stream of > sequential 16k writes is even remotely close to the IO patterns that would > be seen in actual use for MySQL. Most likely in actual use you'll be > seeing a big mix of random reads and writes, and hopefully at least some > parallelism (though this depends on the number of databases, number of > users, and the workload!). > > Ceph is pretty good at small random IO with lots of parallelism on > spinning disk backed OSDs (So long as you use SSD journals or controllers > with WB cache). It's much harder to get native-level IOPS rates with SSD > backed OSDs though. The latency involved in distributing and processing > all of that data becomes a much bigger deal. Having said that, we are > actively working on improving latency as much as we can. :) > > Mark > > > >> >> On Sun, Jun 22, 2014 at 11:50 AM, Mark Kirkwood >> <mark.kirkw...@catalyst.net.nz> wrote: >> >>> On 22/06/14 14:09, Mark Kirkwood wrote: >>> >>> Upgrading the VM to 14.04 and restesting the case *without* direct I get: >>> >>> - 164 MB/s (librbd) >>> - 115 MB/s (kernel 3.13) >>> >>> So managing to almost get native performance out of the librbd case. I >>> tweaked both filestore max and min sync intervals (100 and 10 resp) just >>> to >>> see if I could actually avoid writing to the spinners while the test was >>> in >>> progress (still seeing some, but clearly fewer). >>> >>> However no improvement at all *with* direct enabled. The output of >>> iostat on >>> the host while the direct test is in progress is interesting: >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 11.73 0.00 5.04 0.76 0.00 82.47 >>> >>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s >>> avgrq-sz >>> avgqu-sz await r_await w_await svctm %util >>> sda 0.00 0.00 0.00 11.00 0.00 4.02 749.09 >>> 0.14 12.36 0.00 12.36 6.55 7.20 >>> sdb 0.00 0.00 0.00 11.00 0.00 4.02 749.09 >>> 0.14 12.36 0.00 12.36 5.82 6.40 >>> sdc 0.00 0.00 0.00 435.00 0.00 4.29 20.21 >>> 0.53 1.21 0.00 1.21 1.21 52.80 >>> sdd 0.00 0.00 0.00 435.00 0.00 4.29 20.21 >>> 0.52 1.20 0.00 1.20 1.20 52.40 >>> >>> (sda,b are the spinners sdc,d the ssds). Something is making the journal >>> work very hard for its 4.29 MB/s! >>> >>> regards >>> >>> Mark >>> >>> >>> Leaving >>>> off direct I'm seeing about 140 MB/s (librbd) and 90 MB/s (kernel 3.11 >>>> [2]). The ssd's can do writes at about 180 MB/s each... which is >>>> something to look at another day[1]. >>>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> >> > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
