We just picked up the fastest SSD we could in the local biccamera, which
turned out to be a CSSDーSM32NI, with supposedly 95MB/s write speed.
I put it in place, and replaced the slog over:
**** 0m49.173s
**** 0m48.809s
So, it is slower than the CF test. This is disappointing. Everyone else
seems to use Intel X25-M, which have a write-speed of 170MB/s (2nd
generation) so perhaps that is why it works better for them. It is
curious that it is slower than the CF card. Perhaps because it shares
with so many other SATA devices?
Oh and we'll probably have to get a 3.5" frame for it, as I doubt it'll
stay standing after the next earthquake. :)
Lund
Jorgen Lundman wrote:
This thread started over in nfs-discuss, as it appeared to be an nfs
problem initially. Or at the very least, interaction between nfs and zil.
Just summarising speeds we have found when untarring something. Always
in a new/empty directory. Only looking at write speed. read is always
very fast.
The reason we started to look at this was because the 7 year old netapp
being phased out, could untar the test file in 11 seconds. The
x4500/x4540 Suns took 5 minutes.
For all our tests, we used MTOS-4.261-ja.tar.gz, just a random tarball I
had lying around, but it can be downloaded here if you want the same
test. (http://www.movabletype.org/downloads/stable/MTOS-4.261-ja.tar.gz)
The command executed generally, is:
# mkdir .test34 && time gtar --directory=.test34 -zxf
/tmp/MTOS-4.261-ja.tar.gz
Solaris 10 1/06 intel client: netapp 6.5.1 FAS960 server: NFSv3
**** 0m11.114s
Solaris 10 6/06 intel client: x4500 OpenSolaris svn117 server: nfsv4
**** 5m11.654s
Solaris 10 6/06 intel client: x4500 Solaris 10 10/08 server: nfsv3
**** 8m55.911s
Solaris 10 6/06 intel client: x4500 Solaris 10 10/08 server: nfsv4
**** 10m32.629s
Just untarring the tarball on the x4500 itself:
----------------------------: x4500 OpenSolaris svn117 server
**** 0m0.478s
----------------------------: x4500 Solaris 10 10/08 server
**** 0m1.361s
So ZFS itself is very fast. Replacing NFS with different protocols,
identical setup, just changing tar with rsync, and nfsd with sshd.
The baseline test, using:
"rsync -are ssh /tmp/MTOS-4.261-ja /export/x4500/testXX"
Solaris 10 6/06 intel client: x4500 OpenSolaris svn117 : rsync on nfsv4
**** 3m44.857s
Solaris 10 6/06 intel client: x4500 OpenSolaris svn117 : rsync+ssh
**** 0m1.387s
So, get rid of nfsd and it goes from 3 minutes to 1 second!
Lets share it with smb, and mount it:
OsX 10.5.6 intel client: x4500 OpenSolaris svn117 : smb+untar
**** 0m24.480s
Neat, even SMB can beat nfs in default settings.
This would then indicate to me that nfsd is broken somehow, but then we
try again after only disabling ZIL.
Solaris 10 6/06 : x4500 OpenSolaris svn117 DISABLE ZIL: nfsv4
**** 0m8.453s
**** 0m8.284s
**** 0m8.264s
Nice, so this is theoretically the fastest NFS speeds we can reach? We
run postfix+dovecot for mail, which probably would be safe to not use
ZIL. The other type is FTP/WWW/CGI, which has more active
writes/updates. Probably not as good. Comments?
Enable ZIL, but disable zfscache (Just as a test, I have been told
disabling zfscache is far more dangerous).
Solaris 10 6/06 : x4500 OpenSolaris svn117 DISABLE zfscacheflush: nfsv4
**** 0m45.139s
Interesting. Anyway, enable ZIL and zfscacheflush again, and learn a
whole lot about slog.
First I tried creating a 2G slog on the boot mirror:
Solaris 10 6/06 : x4500 OpenSolaris svn117 slog boot pool: nfsv4
**** 1m59.970s
Some improvements. For a lark, I created a 2GB file in /tmp/ and changed
the slog to that. (I know, having the slog in volatile RAM is pretty
much the same as disabling ZIL. But it should give me theoretical
maximum speed with ZIL enabled right?).
Solaris 10 6/06 : x4500 OpenSolaris svn117 slog /tmp/junk: nfsv4
**** 0m8.916s
Nice! Same speed as ZIL disabled. Since this is a X4540, we thought we
would test with a CF card attached. Alas the 600X (92MB/s) card are not
out until next month, rats! So, we bought a 300X (40MB/s) card.
Solaris 10 6/06 : x4500 OpenSolaris svn117 slog 300X CFFlash: nfsv4
**** 0m26.566s
Not too bad really. But you have to reboot to see a CF card, fiddle with
BIOS for the boot order etc. Just not an easy add on a live system. A
SATA emulated SSD DISK can be hot-swapped.
Also, I learned an interesting lesson about rebooting with slog at
/tmp/junk.
I am hoping to pick up a SSD SATA device today and see what speeds we
get out of that.
That rsync (1s) vs nfs(8s) I can accept as over-head on a much more
complicated protocol, but why would it take 3 minutes to write the same
data on the same pool, with rsync(1s) vs nfs(3m)? The ZIL was on, slog
is default, but both writing the same way. Does nfsd add FD_SYNC to
every close regardless as to whether the application did or not?
This I have not yet wrapped my head around.
For example, I know rsync and tar does not use fdsync (but dovecot does)
on its close(), but does NFS make it fdsync anyway?
Sorry for the giant email.
--
Jorgen Lundman | <lund...@lundman.net>
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
