Hi Brad, comments below...
On Dec 27, 2009, at 10:24 PM, Brad wrote:
Richard - the l2arc is c1t13d0. What tools can be use to show the
l2arc stats?
raidz1 2.68T 580G 543 453 4.22M 3.70M
c1t1d0 - - 258 102 689K 358K
c1t2d0 - - 256 103 684K 354K
c1t3d0 - - 258 102 690K 359K
c1t4d0 - - 260 103 687K 354K
c1t5d0 - - 255 101 686K 358K
c1t6d0 - - 263 103 685K 354K
c1t7d0 - - 259 101 689K 358K
c1t8d0 - - 259 103 687K 354K
c1t9d0 - - 260 102 689K 358K
c1t10d0 - - 263 103 686K 354K
c1t11d0 - - 260 102 687K 359K
c1t12d0 - - 263 104 684K 354K
c1t14d0 396K 29.5G 0 65 7 3.61M
cache - - - - - -
c1t13d0 29.7G 11.1M 157 84 3.93M 6.45M
We've added 16GB to the box bring the overall total to 32GB.
In general, this is always a good idea.
arc_max is set to 8GB:
set zfs:zfs_arc_max = 8589934592
You will be well served to add much more memory to the SGA
and reduce that to the ARC. More below...
arc_summary output:
ARC Size:
Current Size: 8192 MB (arcsize)
Target Size (Adaptive): 8192 MB (c)
Min Size (Hard Limit): 1024 MB (zfs_arc_min)
Max Size (Hard Limit): 8192 MB (zfs_arc_max)
ARC Size Breakdown:
Most Recently Used Cache Size: 39% 3243 MB (p)
Most Frequently Used Cache Size: 60% 4948 MB (c-p)
ARC Efficency:
Cache Access Total: 154663786
Cache Hit Ratio: 41% 64221251 [Defined
State for buffer]
Cache Miss Ratio: 58% 90442535 [Undefined
State for Buffer]
REAL Hit Ratio: 41% 64221251 [MRU/MFU Hits
Only]
Data Demand Efficiency: 38%
Data Prefetch Efficiency: DISABLED (zfs_prefetch_disable)
CACHE HITS BY CACHE LIST:
Anon: --% Counter Rolled.
Most Recently Used: 17% 11118906
(mru) [ Return Customer ]
Most Frequently Used: 82% 53102345
(mfu) [ Frequent Customer ]
Most Recently Used Ghost: 14% 9427708
(mru_ghost) [ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 6% 4344287
(mfu_ghost) [ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 84% 54444108
Prefetch Data: 0% 0
Demand Metadata: 15% 9777143
Prefetch Metadata: 0% 0
CACHE MISSES BY DATA TYPE:
Demand Data: 96% 87542292
Prefetch Data: 0% 0
Demand Metadata: 3% 2900243
Prefetch Metadata: 0% 0
Also disabled file-level pre-fletch and vdev cache max:
set zfs:zfs_prefetch_disable = 1
set zfs:zfs_vdev_cache_max = 0x1
I think this is a waste of time. The database will prefetch,
by default, so you might as well start that work ahead of time.
Note that ZFS uses an intelligent prefetch algorithm, so if it
detects that the accesses are purely random, it won't prefetch.
After reading about some issues with concurrent ios, I tweaked the
setting down from 35 to 1 and it reduced the response times greatly
(2 -> 8ms):
set zfs:zfs_vdev_max_pending=1
This can be a red herring. Judging by the number of IOPS below,
it has not improved. At this point, I will assume you are using
disks that have NCQ or CTQ (eg most SATA and all FC/SAS drives).
If you only issue one command at a time, you effectively disable
NCQ and thus cannot take advantage of its efficiencies.
It did increased the actv...I'm still unsure about the side-effects
here:
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
2295.2 398.7 4.2 7.2 0.0 18.6 0.0 6.9 0 1084 c1
0.0 0.8 0.0 0.0 0.0 0.0 0.0 0.1 0 0 c1t0d0
190.3 22.9 0.4 0.0 0.0 1.5 0.0 7.0 0 87 c1t1d0
180.9 20.6 0.3 0.0 0.0 1.7 0.0 8.5 0 95 c1t2d0
195.0 43.0 0.3 0.2 0.0 1.6 0.0 6.8 0 93 c1t3d0
193.2 21.7 0.4 0.0 0.0 1.5 0.0 6.8 0 88 c1t4d0
195.7 34.8 0.3 0.1 0.0 1.7 0.0 7.5 0 97 c1t5d0
186.8 20.6 0.3 0.0 0.0 1.5 0.0 7.3 0 88 c1t6d0
188.4 21.0 0.4 0.0 0.0 1.6 0.0 7.7 0 91 c1t7d0
189.6 21.2 0.3 0.0 0.0 1.6 0.0 7.4 0 91 c1t8d0
193.8 22.6 0.4 0.0 0.0 1.5 0.0 7.1 0 91 c1t9d0
192.6 20.8 0.3 0.0 0.0 1.4 0.0 6.8 0 88 c1t10d0
195.7 22.2 0.3 0.0 0.0 1.5 0.0 6.7 0 88 c1t11d0
184.7 20.3 0.3 0.0 0.0 1.4 0.0 6.8 0 84 c1t12d0
7.3 82.4 0.1 5.5 0.0 0.0 0.0 0.2 0 1 c1t13d0
1.3 23.9 0.0 1.3 0.0 0.0 0.0 0.2 0 0 c1t14d0
I'm still in talks with the dba in seeing if we can raise the SGA
from 4GB to 6GB to see if it'll help.
Try an SGA more like 20-25 GB. Remember, the database can cache more
effectively than any file system underneath. The best I/O is the I/O
you don't
have to make.
-- richard
The changes that showed a lot of improvement is disabling file/
device level pre-fletch and reducing concurrent ios from 35 to 1
(tried 10 but it didn't help much). Is there anything else that
could be tweaked to increase write performance? Record sizes are
set according to 8K and 128K for redo logs.
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
