Hello
i got a small 3 node ceph cluster and i'm doing some bench marking related to
performance with drive write caching.
the reason i started was because i wanted to test the SSDs i have for their
performance for use as db device for the osds and make sure they are setup as
good as i can get it.
i read that turning off write cache can be beneficial even when it sounds
backwards.
this seems to be true.
i used mainly fio and "iostat -x" to test using something like:
fio --filename=/dev/ceph-db-0/bench --direct=1 --sync=1 --rw=write --bs=4k
--numjobs=5 --iodepth=1 --runtime=60 --time_based --group_reporting
and then testing this with write cache turned off and on to compare the results.
also with and without sync in fio command above.
one thing i observed related to turning off the write cache on drives was that
it appears a reboot is needed for it to have any effect.
and this is where it gets strange and the part i don't get.
the disks i have, seagate nytro sas3 ssd, according to the drive manual the
drive don't care what you set the WCE bit to and it will do write caching
internally regardless.
most likely because it is an enterprise disk with built in power loss
protection.
BUT it makes a big difference to the performance and the flush per seconds in
iostat.
so it appears that if you boot and the drive got its write cache disabled right
from the start (dmesg contains stuff like: "sd 0:0:0:0: [sda] Write cache:
disabled") then linux wont send any flush to the drive and you get good
performance.
if you change the write caching on a drive during runtime (sdparm for sas or
hdparm for sata) then it wont change anything.
why is that? why do i have to do a reboot?
i mean, lets say you boot with write cache disabled, linux decides to never
send flush and you change it after boot to enable the cache, if there is no
flush then you risk your data in case of a power loss, or?
this is not very obvious or good behavior i think (i hope i'm wrong and some
one can enlighten me)
for sas drives sdparm -s WCE=0 --save /dev/sdX appears to do the right thing
and it survives a reboot.
but for sata disks hdparm -W 0 -K 1 /dev/sdX makes the change but as long as
drive is connected to sas controller it still gets the write cache enabled at
boot so i bet sas controller also messes with the write cache setting on the
drives.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
