[zfs-discuss] Un/Expected ZFS performance?

[jiniusatwork-zfs]

I'm running Postgresql (v8.1.10) on Solaris 10 (Sparc) from within a non-global 
zone. I originally had the database "storage" in the non-global zone (e.g. 
/var/local/pgsql/data on a UFS filesystem) and was getting performance of "X" 
(e.g. from a TPC-like application: http://www.tpc.org). I then wanted to try 
relocating the database storage from the zone (UFS filesystem) over to a 
ZFS-based filesystem (where I could do things like set quotas, etc.). When I do 
this, I get roughly half the performance (X/2) I did on the UFS system. 

I 
ran 
some 
low-level 
I/O 
tests 
(from 
http://iozone.org/) 
on 
my 
setup 
and 
have 
listed 
a 
sampling 
below 
for 
an 
8k 
file 
and 
8k 
record 
size:

[Hopefully 
the 
table 
formatting 
survives]

UFS 
filesystem 
[on 
local 
disk]
==
Run  
KB  
reclen  
  
write   
 
rewrite  
  
  
 
read  
   
 
reread  

  
1  
  
  
8  
  
  
  
8  
  
 
40632  
156938  
 
199960  
 
222501  
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-> 
no 
fsync 
include]
  
2  
  
  
8  
  
  
  
8  
  
  
 
4517  

 
5434  
  
 
11997  
  
 
11052  
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-e 
-> 
fsync 
included]
  
3  
  
  
8  
  
  
  
8  
  
  
 
4570  

 5578  
 
199960  
 
215360  
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-o 
-> 
usig 
O_SYNC]

ZFS 
filesystem 
[on 
StorageTek 
2530 
Array 
in 
RAID 
1+0 
configuration 
with 
a 
512K 
segment 
size]
==
Run 
KB  
reclen  
  
write  
 
rewrite  
  
  
 
read  
  
 
reread  
--
  
3  
  
8  
  
  
  
8  
   
 
52281  
  
  
95107  
 
142902  
 
142902 
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-> 
no 
fsync 
include]
  
4  
  
8  
  
  
  
8  
  
   
  
996  
  
  
  
1013  
 
129152  
 
114206 
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-e 
-> 
fsync 
included]
  
5  
  
8  
  
  
  
8  
  
   
  
925  
  
  
  
1007  
 
145379  
 
170495 
[./iozone 
-i 
0 
-i 
1 
-r 
8 
-s 
8 
-o 
-> 
usig 
O_SYNC]

Comparing run 1 and 3 shows that ZFS is roughly 20% faster on (unsynchronized) 
writes versus UFS. What's really surprising, to me at least, is that in cases 3 
and 5, for example,  ZFS becomes almost 400% slower on synchronized writes 
versus UFS. I realize that the ZFS-on-RAID setup has a "safety" penalty, but 
should it really be 400% slower than UFS? If not, then I'm hoping for 
suggestions on how to get some better ZFS performance from this setup.

Thanks,
Bob


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Un/Expected ZFS performance?

[jiniusatwork-zfs]

- Original Message 
> From: Marion Hakanson <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc: zfs-discuss@opensolaris.org
> Sent: Friday, February 1, 2008 1:01:46 PM
> Subject: Re: [zfs-discuss] Un/Expected ZFS performance?
> 
> [EMAIL PROTECTED] said:
> > . . .
> > ZFS filesystem [on StorageTek 2530 Array in RAID 1+0 configuration
> > with a 512K segment size]
> > . . .
> > Comparing run 1 and 3 shows that ZFS is roughly 20% faster on
> > (unsynchronized) writes versus UFS. What's really surprising, to me at 
> > least,
> > is that in cases 3 and 5, for example, ZFS becomes almost 400% slower on
> > synchronized writes versus UFS. I realize that the ZFS-on-RAID setup has a
> > "safety" penalty, but should it really be 400% slower than UFS? If not, then
> > I'm hoping for suggestions on how to get some better ZFS performance from
> > this setup. 
> 
> 
> I don't think there is any "safety penalty" for ZFS on RAID, unless you're
> comparing it to ZFS on JBOD. On RAID without ZFS-level redundancy, you only
> give up ZFS-level self-healing.
> 
> The sync-write issue here is likely similar to that of an NFS server. If all
> of your ZFS pools on this system are on battery-backed cache RAID (e.g. the
> 2530 array), then you could safely set zfs_nocacheflush=1. If not, then
> there should be a way to set the 2530 to ignore the ZFS sync-cache requests.
> 
> Give it a try and let us all know how it affects your tests. We've got
> a 2530 here doing Oracle duty, but it's so much faster than the storage
> it replaced that we haven't bothered doing any performance tuning.
> 
Marion,

Thanks for the reply. As it turns out, I think I'm going to try a different 
strategy. Apparently, I can make the 2530 acknowledge the "sync" requests back 
to the OS without causing the write cache to get flushed (which was happening 
by default). Setting zfs_nocacheflush=1 would, I think, accomplish the same 
thing (i.e. ignore sync requests) from the OS side, but this would end up being 
a "global" ZFS setting on the OS. The problem with the latter solution is that 
if I ever decided to add non-intelligent storage to the system, it wouldn't get 
the sync calls either. 

Thanks again,
Bob

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss