On 05 January, 2007 - Mark Maybee sent me these 1,5K bytes:
> So it looks like this data does not include ::kmastat info from *after*
> you reset arc_reduce_dnlc_percent. Can I get that?
Yeah, attached. (although about 18 hours after the others)
> What I suspect is happening:
> 1 with your large ncsize, you eventually ran the machine out
> of memory because (currently) the arc is not accounting for
> the space consumed by "auxiliary" caches (dnode_t, etc.).
> 2 the arc could not reduce at this point since almost all of
> its memory was tied up by the dnlc refs.
> 3 when you eventually allowed the arc to reduce the dnlc size,
> it managed to free up some space, but much of this did not
> "appear" because it was tied up in slabs in the auxiliary
> caches (fragmentation).
>
> We are working on a fix for number 1 above.
Great!
> You should *not* be setting arc_reduce_dnlc_percent to zero, even if
> you want a large number of dnlc entries. You are tying the arc hands
> here, so it has no ability to reduce its size.
> Number 3 is the most difficult issue. We are looking into that at the
> moment as well.
Any idea where all the memory is going? I sure hope that 500k dnlc
entries (+dnode_t's etc belonging to that) isn't using up about 2GB
ram..?
/Tomas
--
Tomas Ögren, [EMAIL PROTECTED], http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1 16 66933 68072 1097728 1661005 0
kmem_magazine_3 32 5156 185166 5971968 3651757 0
kmem_magazine_7 64 128279 167132 10780672 833454 0
kmem_magazine_15 128 95475 95508 12419072 772265 0
kmem_magazine_31 256 10766 10850 2867200 446593 0
kmem_magazine_47 384 394 1344 524288 268565 0
kmem_magazine_63 512 15 1395 761856 892021 0
kmem_magazine_95 768 136 220 180224 1724 0
kmem_magazine_143 1152 561 567 663552 561 0
kmem_slab_cache 56 31587 57130 3227648 8832570 0
kmem_bufctl_cache 24 259255 275268 6651904 11041532 0
kmem_bufctl_audit_cache 128 0 0 0 0 0
kmem_va_8192 8192 156227 182080 1491599360 563808 0
kmem_va_16384 16384 6760 24016 393478144 2608303 0
kmem_va_24576 24576 30 90 2359296 26820 0
kmem_va_32768 32768 4102 9864 323223552 8220474 0
kmem_va_40960 40960 38 264 11534336 237265 0
kmem_va_49152 49152 5 20 1048576 3009 0
kmem_va_57344 57344 98 428 28049408 543538 0
kmem_va_65536 65536 477 2992 196083712 835607 0
kmem_alloc_8 8 8697 13221 106496 7731684 0
kmem_alloc_16 16 9269 12700 204800 562429206 0
kmem_alloc_24 24 8622 13221 319488 148101813 0
kmem_alloc_32 32 32929 163322 5267456 673202709 0
kmem_alloc_40 40 44236 55622 2244608 57956587 0
kmem_alloc_48 48 10631 62361 3022848 66508294 0
kmem_alloc_56 56 16033 96570 5455872 3931601 0
kmem_alloc_64 64 18570 157607 10166272 46889692 0
kmem_alloc_80 80 135312 622564 50495488 40453962 0
kmem_alloc_96 96 179070 291396 28418048 23482483 0
kmem_alloc_112 112 50004 135072 15368192 98592237 0
kmem_alloc_128 128 11253 23058 2998272 8022665 0
kmem_alloc_160 160 3172 7800 1277952 1022154 0
kmem_alloc_192 192 298 924 180224 2142183 0
kmem_alloc_224 224 279 360 81920 1391779 0
kmem_alloc_256 256 7535 24800 6553600 188838186 0
kmem_alloc_320 320 1689 1750 573440 560146 0
kmem_alloc_384 384 274 1827 712704 60590097 0
kmem_alloc_448 448 2706 2772 1261568 312999151 0
kmem_alloc_512 512 316 690 376832 16436341 0
kmem_alloc_640 640 88 12732 8691712 517476456 0
kmem_alloc_768 768 23 40 32768 180648 0
kmem_alloc_896 896 27 36 32768 307859 0
kmem_alloc_1152 1152 721 798 933888 3858662 0
kmem_alloc_1344 1344 55 66 90112 451770 0
kmem_alloc_1600 1600 30 40 65536 3009471 0
kmem_alloc_2048 2048 98 120 245760 2764046 0
kmem_alloc_2688 2688 103 114 311296 762011 0
kmem_alloc_4096 4096 53 58 237568 173305 0
kmem_alloc_8192 8192 5157 5353 43851776 1335040 0
kmem_alloc_12288 12288 15 32 393216 14583 0
kmem_alloc_16384 16384 38 39 638976 2034 0
streams_mblk 64 18468 18542 1196032 530378743 0
streams_dblk_16 128 55 126 16384 779280 0
streams_dblk_80 192 292 462 90112 4522693 0
streams_dblk_144 256 2 62 16384 489060682 0
streams_dblk_208 320 265 375 122880 251539 0
streams_dblk_272 384 7 84 32768 97143 0
streams_dblk_336 448 0 18 8192 6065 0
streams_dblk_528 640 0 24 16384 21748 0
streams_dblk_1040 1152 1 35 40960 18605 0
streams_dblk_1488 1600 0 10 16384 5225 0
streams_dblk_1936 2048 1282 17092 35004416 352153376 0
streams_dblk_2576 2688 0 12 32768 2211458 0
streams_dblk_3920 4032 0 4 16384 8864 0
streams_dblk_8192 112 0 63 8192 94593 0
streams_dblk_12112 12224 0 4 49152 46740 0
streams_dblk_16384 112 0 63 8192 96031 0
streams_dblk_20304 20416 0 4 81920 28034 0
streams_dblk_24576 112 0 63 8192 50449 0
streams_dblk_28496 28608 0 4 114688 10747 0
streams_dblk_32768 112 0 126 16384 19959114 0
streams_dblk_36688 36800 0 38 1400832 928268 0
streams_dblk_40960 112 0 63 8192 1110 0
streams_dblk_44880 44992 0 4 180224 808 0
streams_dblk_49152 112 0 63 8192 1550 0
streams_dblk_53072 53184 0 4 212992 1301 0
streams_dblk_57344 112 0 63 8192 620 0
streams_dblk_61264 61376 0 4 245760 513 0
streams_dblk_65536 112 0 63 8192 19819 0
streams_dblk_69456 69568 0 4 278528 1171 0
streams_dblk_73728 112 0 63 8192 304 0
streams_dblk_esb 112 0 126 16384 20804 0
streams_fthdr 264 0 0 0 0 0
streams_ftblk 232 0 0 0 0 0
multidata 248 0 0 0 0 0
multidata_pdslab 7112 0 0 0 0 0
multidata_pattbl 32 0 0 0 0 0
log_cons_cache 48 8 169 8192 41 0
taskq_ent_cache 56 2475 3480 196608 27572807 0
taskq_cache 216 70 111 24576 92 0
id32_cache 32 0 0 0 0 0
bp_map_16384 16384 0 32 524288 762 0
bp_map_32768 32768 0 16 524288 2563 0
bp_map_49152 49152 0 10 524288 2211 0
bp_map_65536 65536 0 8 524288 1874 0
bp_map_81920 81920 0 6 524288 1705 0
bp_map_98304 98304 0 5 524288 1409 0
bp_map_114688 114688 0 4 524288 1092 0
bp_map_131072 131072 0 4 524288 1444 0
memseg_cache 112 0 0 0 0 0
mod_hash_entries 24 249 678 16384 4185 0
ipp_mod 304 0 0 0 0 0
ipp_action 368 0 0 0 0 0
ipp_packet 64 0 0 0 0 0
mmuctxdom_cache 192 2 42 8192 2 0
sfmmuid_cache 184 49 84 16384 22032 0
sfmmu_tsbinfo_cache 64 48 127 8192 49826 0
sfmmu_tsb8k_cache 8192 0 0 0 0 0
sfmmu_tsb_cache 8192 28 35 286720 24083 0
sfmmu8_cache 312 39105 87360 27525120 8086400 0
sfmmu1_cache 88 3557 3588 319488 193984 0
pa_hment_cache 64 43 127 8192 27925 0
ism_blk_cache 272 0 0 0 0 0
ism_ment_cache 32 0 0 0 0 0
seg_cache 72 2892 3390 245760 2149118 0
dev_info_node_cache 488 174 320 163840 1217 0
segkp_8192 8192 42 48 393216 11700 0
segkp_16384 16384 0 0 0 0 0
segkp_24576 24576 0 0 0 0 0
segkp_32768 32768 547 600 19660800 4234 0
segkp_40960 40960 0 0 0 0 0
umem_np_8192 8192 0 32 262144 222 0
umem_np_16384 16384 0 16 262144 100 0
umem_np_24576 24576 0 0 0 0 0
umem_np_32768 32768 0 8 262144 9 0
umem_np_40960 40960 0 0 0 0 0
umem_np_49152 49152 0 0 0 0 0
umem_np_57344 57344 0 0 0 0 0
umem_np_65536 65536 0 4 262144 9 0
thread_cache 800 231 260 212992 29167 0
wbuf32_cache 512 228 270 147456 29187 0
wbuf64_cache 1024 3 14 16384 738 0
lwp_cache 912 231 264 270336 1136 0
turnstile_cache 64 541 635 40960 34817 0
tslabel_cache 48 2 169 8192 2 0
cred_cache 172 91 138 24576 2574 0
rctl_cache 40 741 1015 40960 145396 0
rctl_val_cache 64 1454 1905 122880 317132 0
task_cache 104 34 78 8192 410 0
cyclic_id_cache 64 10 127 8192 31 0
dnlc_space_cache 24 19 339 8192 6062 0
vn_cache 240 405388 657696 173801472 948191 0
vsk_anchor_cache 40 15 203 8192 44 0
file_cache 56 402 580 32768 827834 0
stream_head_cache 368 195 220 81920 74333 0
queue_cache 656 508 540 368640 212667 0
syncq_cache 160 11 50 8192 29 0
qband_cache 64 2 127 8192 2 0
linkinfo_cache 48 4 169 8192 4 0
ciputctrl_cache 128 4 63 8192 4 0
serializer_cache 64 26 127 8192 151 0
as_cache 224 48 72 16384 22031 0
marker_cache 128 0 63 8192 156031 0
anon_cache 48 15062 16224 786432 3343368 0
anonmap_cache 64 1754 2159 139264 437468 0
segvn_cache 104 2892 3432 360448 1072573 0
flk_edges 48 0 169 8192 1 0
fdb_cache 104 0 0 0 0 0
timer_cache 136 1 59 8192 2 0
physio_buf_cache 248 0 32 8192 305 0
snode_cache 152 363 424 65536 290321 0
ufs_inode_cache 368 58473 75042 27942912 118311 0
directio_buf_cache 272 0 0 0 0 0
lufs_save 24 0 339 8192 45759 0
lufs_bufs 256 0 31 8192 47055 0
lufs_mapentry_cache 112 44 1656 188416 1067289 0
pcisch0_dvma_8192 8192 44 1008 8257536 269346609 0
pcisch1_dvma_8192 8192 1282 1328 10878976 312538496 0
fctl_cache 112 0 72 8192 9 0
fp1_cache 728 1 11 8192 6 0
fcp1_cache 1224 2 39 49152 1440466 0
dv_node_cache 128 93 189 24576 839 0
sdev_node_cache 200 582 600 122880 762 0
clnt_clts_endpnt_cache 88 2 92 8192 190 0
md_stripe_parent 96 0 252 24576 1269327 0
md_stripe_child 312 0 260 81920 1398467 0
md_mirror_parent 160 0 250 40960 803301 0
md_mirror_child 304 0 260 81920 1397128 0
md_mirror_wow 16440 0 8 139264 18143 0
kcf_sreq_cache 48 0 0 0 0 0
kcf_areq_cache 272 0 0 0 0 0
kcf_context_cache 88 0 0 0 0 0
ipsec_actions 72 0 113 8192 1920 0
ipsec_selectors 80 0 0 0 0 0
ipsec_policy 72 0 0 0 0 0
ipsec_info 336 0 24 8192 2587 0
ip_minor_arena_1 1 83 128 128 53719 0
ipcl_conn_cache 488 39 60 32768 12549 0
ipcl_tcpconn_cache 1696 47 72 131072 83856 0
radix_mask 32 1 254 8192 1 0
radix_node 112 2 72 8192 2 0
rt_entry 160 3 50 8192 4 0
ire_cache 344 56 92 32768 2534 0
tcp_timercache 88 117 184 16384 13129 0
tcp_sack_info_cache 80 19 101 8192 40546 0
tcp_iphc_cache 120 46 134 16384 82242 0
squeue_cache 136 2 42 8192 2 0
sctp_conn_cache 2264 1 7 16384 1 0
sctp_faddr_cache 168 0 0 0 0 0
sctp_set_cache 24 0 0 0 0 0
sctp_ftsn_set_cache 16 0 0 0 0 0
ire_gw_secattr_cache 32 0 0 0 0 0
sctpsock 616 0 0 0 0 0
sctp_assoc 64 0 0 0 0 0
socktpi_cache 456 20 34 16384 3523 0
socktpi_unix_cache 456 14 34 16384 124 0
mac_impl_cache 384 0 0 0 0 0
dls_cache 152 0 0 0 0 0
soft_ring_cache 176 0 0 0 0 0
dls_vlan_cache 48 0 0 0 0 0
dls_link_cache 368 0 0 0 0 0
dld_ctl_1 1 0 0 0 0 0
drv_secobj_cache 296 0 0 0 0 0
dld_str_cache 272 0 29 8192 1 0
udp_cache 416 34 54 24576 12535 0
process_cache 3080 50 60 196608 11452 0
exacct_object_cache 40 0 0 0 0 0
ch_private_cache 13200 2 2 32768 2 0
hal0_cache 464 0 17 8192 1 0
kssl_cache 1560 0 0 0 0 0
tl_cache 432 39 54 24576 393 0
keysock_1 1 0 0 0 0 0
spdsock_1 1 0 64 64 1 0
fnode_cache 176 5 42 8192 34 0
pipe_cache 320 24 50 16384 9219 0
namefs_inodes_1 1 19 64 64 49 0
port_cache 80 3 101 8192 3 0
ip_minor_1 1 0 0 0 0 0
ar_minor_1 1 0 0 0 0 0
lnode_cache 32 2 254 8192 2 0
zio_buf_512 512 137801 294975 161095680 43660052 0
zio_buf_1024 1024 7 80 81920 7730712 0
zio_buf_1536 1536 4 20 32768 4319811 0
zio_buf_2048 2048 6 16 32768 840840 0
zio_buf_2560 2560 1 12 32768 437525 0
zio_buf_3072 3072 7 16 49152 658297 0
zio_buf_3584 3584 2 9 32768 647863 0
zio_buf_4096 4096 2 6 24576 523423 0
zio_buf_5120 5120 3 16 81920 3124059 0
zio_buf_6144 6144 2 8 49152 857598 0
zio_buf_7168 7168 1 336 2408448 104619346 0
zio_buf_8192 8192 2 5 40960 133985 0
zio_buf_10240 10240 2 8 81920 99216 0
zio_buf_12288 12288 0 6 73728 41462 0
zio_buf_14336 14336 0 116 1662976 13512747 0
zio_buf_16384 16384 6692 6697 109723648 5877279 0
zio_buf_20480 20480 1 20 409600 8538438 0
zio_buf_24576 24576 1 5 122880 12157 0
zio_buf_28672 28672 1 42 1204224 7455093 0
zio_buf_32768 32768 3701 4101 134381568 35395387 0
zio_buf_40960 40960 1 22 901120 1086580 0
zio_buf_49152 49152 3 5 245760 20708 0
zio_buf_57344 57344 2 4 229376 20091 0
zio_buf_65536 65536 365 477 31260672 2831209 0
zio_buf_73728 73728 0 3 221184 12129 0
zio_buf_81920 81920 0 3 245760 7014 0
zio_buf_90112 90112 0 4 360448 5552 0
zio_buf_98304 98304 0 3 294912 5997 0
zio_buf_106496 106496 0 3 319488 9767 0
zio_buf_114688 114688 0 2 229376 4570 0
zio_buf_122880 122880 0 3 368640 3911 0
zio_buf_131072 131072 0 2 262144 136994 0
dmu_buf_impl_t 328 145260 622392 212443136 65461261 0
dnode_t 640 137799 512508 349872128 37995548 0
arc_buf_hdr_t 144 21457 43344 6340608 35018385 0
arc_buf_t 40 10823 19285 778240 60847028 0
zil_lwb_cache 200 0 0 0 0 0
zfs_znode_cache 200 137763 568040 116334592 35683478 0
mpt1_cache 536 7 315 172032 93673786 0
mpt0_cache 536 7 315 172032 92907412 0
md_raid_parent 120 0 0 0 0 0
md_raid_child 1040 0 0 0 0 0
md_raid_cbufs 376 0 0 0 0 0
md_trans_parent 80 0 0 0 0 0
md_trans_child 248 0 0 0 0 0
glm0_cache 472 4 17 8192 1294 0
icmp_minor_1 1 0 0 0 0 0
authkern_cache 72 0 113 8192 32203 0
authloopback_cache 72 0 0 0 0 0
authdes_cache_handle 80 0 0 0 0 0
rnode_cache 648 4 144 98304 16551 0
nfs_access_cache 56 1 1885 106496 2440 0
client_handle_cache 32 4 254 8192 43 0
rnode4_cache 960 0 0 0 0 0
svnode_cache 40 0 0 0 0 0
nfs4_access_cache 56 0 0 0 0 0
client_handle4_cache 32 0 0 0 0 0
nfs4_ace4vals_cache 48 0 0 0 0 0
nfs4_ace4_list_cache 264 0 0 0 0 0
NFS_idmap_cache 48 0 169 8192 6 0
nfslog_small_rec 512 0 0 0 0 0
nfslog_medium_rec 8192 0 0 0 0 0
nfslog_large_rec 32768 0 0 0 0 0
exi_cache_handle 40 19 203 8192 59 0
dtrace_state_cache 2048 0 4 8192 9 0
lm_vnode 184 0 0 0 0 0
lm_xprt 32 0 0 0 0 0
lm_sysid 160 0 0 0 0 0
lm_client 128 0 0 0 0 0
lm_async 32 0 0 0 0 0
lm_sleep 96 0 0 0 0 0
lm_config 80 2 101 8192 2 0
pty_map 64 3 127 8192 7 0
glm1_cache 472 0 17 8192 152 0
Client_entry_cache 744 0 10 8192 1 0
OpenOwner_entry_cache 528 0 0 0 0 0
OpenStateID_entry_cache 256 0 0 0 0 0
LockStateID_entry_cache 504 0 0 0 0 0
Lockowner_entry_cache 176 0 0 0 0 0
File_entry_cache 288 0 0 0 0 0
DelegStateID_entry_cache 248 0 0 0 0 0
md_softpart_parent 88 0 0 0 0 0
md_softpart_child 304 0 0 0 0 0
fcip1_cache 728 0 0 0 0 0
fcip1_sendup_cache 24 0 0 0 0 0
fcsm_job_cache 104 0 0 0 0 0
fcsm1_cmd_cache 712 0 0 0 0 0
crypto_session_cache 96 0 0 0 0 0
aggr_port_cache 1056 0 0 0 0 0
aggr_grp_cache 680 0 0 0 0 0
------------------------- ------ ------ ------ --------- --------- -----
Total [static] 434176 109051 0
Total [hat_memload] 27525120 8086400 0
Total [kmem_msb] 45146112 28402047 0
Total [kmem_va] 2447376384 13038824 0
Total [kmem_default] 1569161216 667252352 0
Total [bp_map] 4194304 13060 0
Total [kmem_tsb_default] 286720 24083 0
Total [hat_memload1] 319488 193984 0
Total [umem_np] 1048576 340 0
Total [segkp] 20054016 15934 0
Total [pcisch0_dvma] 8257536 269346609 0
Total [pcisch1_dvma] 10878976 312538496 0
Total [ip_minor_arena] 128 53719 0
Total [spdsock] 64 1 0
Total [namefs_inodes] 64 49 0
------------------------- ------ ------ ------ --------- --------- -----
vmem memory memory memory alloc alloc
name in use total import succeed fail
------------------------- --------- ---------- --------- --------- -----
heap 2632138752 4398046511104 0 57256 0
vmem_metadata 33038336 33292288 33292288 3882 0
vmem_seg 31260672 31260672 31260672 3816 0
vmem_hash 1450752 1466368 1466368 74 0
vmem_vmem 295800 346096 311296 101 0
static 458752 458752 458752 55 0
static_alloc 8320 16384 16384 3 0
hat_memload 27525120 27525120 27525120 3434 0
kstat 266568 278528 212992 944 0
kmem_metadata 48144384 48234496 48234496 6878 0
kmem_msb 45146112 45146112 45146112 6559 0
kmem_cache 202872 221184 221184 380 0
kmem_hash 2770944 2777088 2777088 1638 0
kmem_log 131488 139264 139264 6 0
kmem_firewall_va 37429248 37429248 37429248 23998 0
kmem_firewall 0 0 0 0 0
kmem_oversize 36777327 37429248 37429248 24004 0
mod_sysfile 892 8192 8192 25 0
kmem_va 2452135936 2452135936 2452135936 27933 0
kmem_default 1569243136 1569243136 1569243136 8579884 0
little_endian 179168 204800 204800 487 0
big_endian 4455340 4931584 4931584 6631 0
bp_map 4194304 4194304 4194304 713 0
ksyms 1717794 1744896 1744896 305 0
ctf 205377 212992 212992 307 0
kmem_tsb 4194304 4194304 4194304 1 0
kmem_tsb_default 745472 4194304 4194304 5965 0
hat_memload1 319488 319488 319488 111 0
umem_np 1179648 1179648 1179648 12 0
heap32 2086656 134217728 0 506 0
id32 0 0 0 0 0
module_data 1889384 2293760 2031616 399 0
promplat 0 0 0 464 0
trapstat 0 0 0 0 0
heaptext 39337984 134217728 0 122 0
module_text 7705380 7831552 5775360 303 0
logminor_space 24 262137 0 57 0
taskq_id_arena 34 2147483647 0 56 0
segkp 20054016 2147483648 0 2466 0
rctl_ids 32 32767 0 32 0
zoneid_space 0 9998 0 0 0
taskid_space 34 999999 0 410 0
pool_ids 0 999998 0 0 0
contracts 36 2147483646 0 446 0
regspec 884736 5368709120 0 144 0
pcisch0_dvma 8257536 1040187392 0 121529 0
pcisch1_dvma 15728640 1040187392 0 73106 0
ip_minor_arena 128 262140 0 2 0
dld_ctl 0 4294967295 0 0 0
dld_minor_arena 1 4294967295 0 1 0
tl_minor_space 39 262138 0 367 0
keysock 0 4294967295 0 0 0
spdsock 64 4294967295 0 1 0
namefs_inodes 64 65536 0 1 0
ip_minor 0 262142 0 0 0
ar_minor 0 262142 0 0 0
devfsadm_event_channel 1 101 0 1 0
devfsadm_event_channel 1 2 0 1 0
syseventconfd_event_channel 0 101 0 0 0
syseventconfd_event_channel 1 2 0 1 0
syseventd_channel 4 101 0 4 0
syseventd_channel 1 2 0 1 0
icmp_minor 0 262142 0 0 0
lmsysid_space 1 16383 0 1 0
dtrace 948 4294967295 0 61355 0
dtrace_minor 0 4294967293 0 9 0
ptms_minor 3 16 0 7 0
heaptext_holesrc_15 368640 2097152 0 27 0
heaptext_hole_15 320364 368640 368640 72 0
heaptext_holesrc_14 540672 2097152 0 14 0
heaptext_hole_14 501880 540672 540672 33 0
Client_id_space 0 131072 0 1 0
OpenOwner_id_space 0 1048576 0 0 0
OpenStateID_id_space 0 1048576 0 0 0
LockStateID_id_space 0 1048576 0 0 0
Lockowner_id_space 0 1048576 0 0 0
DelegStateID_id_space 0 1048576 0 0 0
logdmux_minor 0 256 0 0 0
heaptext_holesrc_13 0 2097152 0 1 0
heaptext_hole_13 0 0 0 1 0
module_text_holesrc 1409024 4194304 0 28 0
heaptext_hole_0 1374100 1409024 1409024 68 0
heaptext_holesrc_16 557056 2097152 0 16 0
heaptext_hole_16 527752 557056 557056 48 0
crypto 0 16 0 3 0
------------------------- --------- ---------- --------- --------- -----
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
