fio /dev/rbd0 sync=1 has no problem. Doesn't find 'sync cache code' in linux rbd block driver and radosgw api. Seems sync cache is just the concept of librbd (for rbd cache). Just my concerns.
2016-02-26 17:30 GMT+08:00 Huan Zhang <huan.zhang...@gmail.com>: > Hi Nick, > DB's IO pattern depends on config, mysql for example. > innodb_flush_log_at_trx_commit =1, mysql will sync after one transcation. > like: > write > sync > wirte > sync > ... > > innodb_flush_log_at_trx_commit = 5, > write > write > write > write > write > sync > > innodb_flush_log_at_trx_commit = 0, > write > write > ... > one second later. > sync. > > > may not very accurate, but more or less. > > We test mysql tps, with nnodb_flush_log_at_trx_commit =1, get very poor > performance even if we can reach very high O_DIRECT randwrite iops with fio. > > > > > 2016-02-26 16:59 GMT+08:00 Nick Fisk <n...@fisk.me.uk>: > >> > -----Original Message----- >> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf >> Of >> > Huan Zhang >> > Sent: 26 February 2016 06:50 >> > To: Jason Dillaman <dilla...@redhat.com> >> > Cc: josh durgin <josh.dur...@inktank.com>; Nick Fisk <n...@fisk.me.uk>; >> > ceph-users <ceph-us...@ceph.com> >> > Subject: Re: [ceph-users] Guest sync write iops so poor. >> > >> > rbd engine with fsync=1 seems stuck. >> > Jobs: 1 (f=1): [w(1)] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta >> > 1244d:10h:39m:18s] >> > >> > But fio using /dev/rbd0 sync=1 direct=1 ioengine=libaio iodepth=64, get >> very >> > high iops ~35K, similar to direct wirte. >> > >> > I'm confused with that result, IMHO, ceph could just ignore the sync >> cache >> > command since it always use sync write to journal, right? >> >> Even if the data is not sync'd to the data storage part of the OSD, the >> data still has to be written to the journal and this is where the >> performance limit lies. >> >> The very nature of SDS means that you are never going to achieve the same >> latency as you do to a local disk as even if the software side introduced >> no extra latency, just the network latency will severely limit your sync >> performance. >> >> Do you know the IO pattern the DB's generate? I know you can switch most >> DB's to flush with O_DIRECT instead of sync, it might be this helps in your >> case. >> >> Also check out the tech talk from last month about high performance >> databases on Ceph. The presenter gave the impression that, at least in >> their case, not every write was a sync IO. So your results could possibly >> matter less than you think. >> >> Also please search the lists and past presentations about reducing write >> latency. There are a few things you can do like disabling logging and some >> kernel parameters to stop the CPU's entering sleep states/reducing >> frequency. One thing I witnessed that if the Ceph cluster is only running >> at low queue depths, so it's only generating low cpu load, all the cores on >> the CPU's throttle themselves down to their lowest speeds, which really >> hurts latency. >> >> > >> > Why we get so bad sync iops, how ceph handle it? >> > Very appreciated to get your reply! >> > >> > 2016-02-25 22:44 GMT+08:00 Jason Dillaman <dilla...@redhat.com>: >> > > 35K IOPS with ioengine=rbd sounds like the "sync=1" option doesn't >> > actually >> > > work. Or it's not touching the same object (but I wonder whether write >> > > ordering is preserved at that rate?). >> > >> > The fio rbd engine does not support "sync=1"; however, it should support >> > "fsync=1" to accomplish roughly the same effect. >> > >> > Jason >> >> >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
