This thread started with the notion that ZFS provided a very bad NFS service to the point of > 10X degradation over say UFS. What I hope we have an agreement on, is that these scale of performance difference does not come from ZFS but from an NFS service that would sacrifice integrity. Enabling the write-cache (UFS) or disabling the zil (ZFS) are 2 ways to get such speedup.
Here, you have a fair comparison of UFS/ZFS serving NFS and that shows a 10% effect. It would be nice to analyze where that difference comes from. -r Matt Sweeney writes: > Bill, > > I did the same test on the Thumper I'm working on with the NFS vols > converted from ZFS stripes to SVM stripes. In both cases same > number/type of disks in the stripe. In my very simple test ,time for > file in frame*; do cp /inq/$file /outq/$file; done, UFS did > approximately 64 MB/s, the best run for ZFS was approx 58 MB/s. Not a > huge difference for sure, but enough to make you think about switching. > This was single stream over a 10GE link. (x4600 mounting vols from an x4500) > > Matt > > Bill Moore wrote: > > On Thu, Nov 23, 2006 at 03:37:33PM +0100, Roch - PAE wrote: > > > >> Al Hopper writes: > >> > Hi Roch - you are correct in that the data presented was incomplete. I > >> > did'nt present data for the same test with an NFS mount from the same > >> > server, for a UFS based filesystem. So here is that data point: > >> > > >> > $ ptime gunzip -c /tmp/emacs-21.4a.tar.gz |tar xf - > >> > > >> > real 12.671 > >> > user 2.356 > >> > sys 0.228 > >> > > >> > This test is not totally fair, in that the UFS filesystem being shared > >> is > >> > on a single 400Gb SATA drive being used as the boot device - versus the > >> > 5-way raidz config which consists of 5 of those same 400Gb SATA drives. > >> > But the data clearly shows the NFS/ZFS is a bad combination: 2 minutes > >> 33 > >> > Seconds for NFS/ZFS versus 13 Seconds (rouding up) for NFS/UFS. > >> > >> I'd put 100$ on the table that the WCE is enabled on the > >> SATA drive backing UFS. Even if format says it's not, are > >> there not some drives which just ignore the WC disable > >> commands ? > >> > > > > I agree with Roch here. With UFS, if WCE is enabled on the drives > > (which I'm sure it is on Al's SATA drives), UFS is fooled into thinking > > that when it writes a block to disk, it's safe. The drive returns from > > the write amazingly fast (since the data only landed in cache - not the > > media), so you get quick turnarounds (low latency) on NFS, which is the > > only thing that matters on single-threaded performance. > > > > With ZFS, on the other hand, not only do we write data to the drive when > > NFS tells us to, but we issue a DKIOCFLUSHWRITECACHE ioctl to the > > underlying device (FLUSH_CACHE on ATA, SYNCHRONIZE_CACHE on SCSI) to > > ensure that the data that's supposed to be on the disk is really, truly > > on the disk. This takes typically around 4-6ms, which is quite a while. > > Again, this dictates the single-threaded NFS performance. > > > > If you want an apples-to-apples comparison, either try the UFS/ZFS tests > > on a drive that has WCE disabled, or turn off the ZIL on a drive that > > has WCE enabled. I'll bet the difference will be rather slight, perhaps > > in favor of ZFS. > > > > > > --Bill > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss@opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> > <html> > <head> > <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type"> > <title></title> > </head> > <body bgcolor="#ffffff" text="#000000"> > Bill,<br> > <br> > I did the same test on the Thumper I'm working on with the NFS vols > converted from ZFS stripes to SVM stripes. In both cases same > number/type of disks in the stripe. In my very simple test ,time > for > file in frame*; do cp /inq/$file /outq/$file; done, UFS did > approximately 64 MB/s, the best run for ZFS was approx 58 MB/s. Not a > huge difference for sure, but enough to make you think about > switching. This was single stream over a 10GE link. (x4600 mounting > vols from an x4500)<br> > <br> > Matt<br> > <br> > Bill Moore wrote: > <blockquote cite="[EMAIL PROTECTED]" type="cite"> > <pre wrap="">On Thu, Nov 23, 2006 at 03:37:33PM +0100, Roch - PAE wrote: > </pre> > <blockquote type="cite"> > <pre wrap="">Al Hopper writes: > > Hi Roch - you are correct in that the data presented was incomplete. I > > did'nt present data for the same test with an NFS mount from the same > > server, for a UFS based filesystem. So here is that data point: > > > > $ ptime gunzip -c /tmp/emacs-21.4a.tar.gz |tar xf - > > > > real 12.671 > > user 2.356 > > sys 0.228 > > > > This test is not totally fair, in that the UFS filesystem being shared > is > > on a single 400Gb SATA drive being used as the boot device - versus the > > 5-way raidz config which consists of 5 of those same 400Gb SATA drives. > > But the data clearly shows the NFS/ZFS is a bad combination: 2 minutes > 33 > > Seconds for NFS/ZFS versus 13 Seconds (rouding up) for NFS/UFS. > > I'd put 100$ on the table that the WCE is enabled on the > SATA drive backing UFS. Even if format says it's not, are > there not some drives which just ignore the WC disable > commands ? > </pre> > </blockquote> > <pre wrap=""><!----> > I agree with Roch here. With UFS, if WCE is enabled on the drives > (which I'm sure it is on Al's SATA drives), UFS is fooled into thinking > that when it writes a block to disk, it's safe. The drive returns from > the write amazingly fast (since the data only landed in cache - not the > media), so you get quick turnarounds (low latency) on NFS, which is the > only thing that matters on single-threaded performance. > > With ZFS, on the other hand, not only do we write data to the drive when > NFS tells us to, but we issue a DKIOCFLUSHWRITECACHE ioctl to the > underlying device (FLUSH_CACHE on ATA, SYNCHRONIZE_CACHE on SCSI) to > ensure that the data that's supposed to be on the disk is really, truly > on the disk. This takes typically around 4-6ms, which is quite a while. > Again, this dictates the single-threaded NFS performance. > > If you want an apples-to-apples comparison, either try the UFS/ZFS tests > on a drive that has WCE disabled, or turn off the ZIL on a drive that > has WCE enabled. I'll bet the difference will be rather slight, perhaps > in favor of ZFS. > > > --Bill > _______________________________________________ > zfs-discuss mailing list > <a class="moz-txt-link-abbreviated" > href="mailto:zfs-discuss@opensolaris.org";>zfs-discuss@opensolaris.org</a> > <a class="moz-txt-link-freetext" > href="http://mail.opensolaris.org/mailman/listinfo/zfs-discuss";>http://mail.opensolaris.org/mailman/listinfo/zfs-discuss</a> > </pre> > </blockquote> > <br> > </body> > </html> _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
