Robert,
I would be interested in seeing your crash dump. ZFS will consume much
of your memory *in the absence of memory pressure*, but it should be
responsive to memory pressure, and give up memory when this happens. It
looks like you have 8GB of memory on your system? ZFS should never
consume more than 7GB of this under any circumstances. Note there are
a few outstanding bugs that could be coming into play here:
6456888 zpool scrubbing leads to memory exhaustion and system hang
6416757 zfs could still use less memory
6447701 ZFS hangs when iSCSI Target attempts to initialize its backing store
-Mark
P.S. It would be useful to see the output of:
> arc::print
and
> ::kmastat
Robert Milkowski wrote:
Hi.
v440, S10U2 + patches
OS and Kernel Version: SunOS XXXXX 5.10 Generic_118833-20 sun4u sparc
SUNW,Sun-Fire-V440
NFS server with ZFS as a local storage.
We were rsyncing UFS filesystem to ZFS filesystem exported over NFS. After some
time server which exports ZFS over NFS was unresponsive. Operator decided to
force panic and reboot server. Further examination showed that system was
heavily paging probably due to ZFS as no other services are running there.
I had just another problem - looks similar to last one.
I decided to put nfsd into RT class.
I guess ZFS is using all memory for its caches and after some time it fails to
free it and forces system to paging. This is BAD, really BAD.
More details to previous problem.
bash-3.00# savecore /f3-1/
System dump time: Sat Sep 2 03:31:18 2006
Constructing namelist /f3-1//unix.0
Constructing corefile /f3-1//vmcore.0
100% done: 1043993 of 1043993 pages saved
bash-3.00# cd /f3-1/
bash-3.00#
bash-3.00# mdb 0
Loading modules: [ unix krtld genunix dtrace specfs ufs sd md ip sctp usba fcp
fctl qlc ssd lofs zfs random logindmux ptm
cpc nfs ipc ]
::status
debugging crash dump vmcore.0 (64-bit) from XXXXXX
operating system: 5.10 Generic_118833-20 (sun4u)
panic message: sync initiated
dump content: kernel pages only
::spa
ADDR STATE NAME
0000060001271680 ACTIVE f3-1
0000060003bd4dc0 ACTIVE f3-2
::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 1016199 7939 98%
Anon 4420 34 0%
Exec and libs 736 5 0%
Page cache 36 0 0%
Free (cachelist) 1962 15 0%
Free (freelist) 18338 143 2%
Total 1041691 8138
Physical 1024836 8006
::swapinfo
ADDR VNODE PAGES FREE NAME
00000600034ab5a0 600012ff8c0 1048763 1028489 /dev/md/dsk/d15
We were synchronizing lot of small files over nfs and writing to f3-1/d611. I
would say that with ZFS it's expected to be
on low memory most of the time but not to the point when host starts to paging.
bash-3.00# sar -g
SunOS XXXXX 5.10 Generic_118833-20 sun4u 09/02/2006
00:00:00 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
[...]
02:15:01 0.03 0.04 0.02 0.00 0.00
02:20:00 0.04 0.04 0.02 0.00 0.00
02:25:00 0.02 0.03 0.01 0.00 0.00
02:30:00 0.02 0.03 0.01 0.00 0.00
02:35:00 0.03 0.03 0.01 0.00 0.00
02:40:01 0.03 0.04 0.03 0.00 0.00
02:45:02 5.98 82.77 93.20 65115.59 0.00
03:39:28 unix restarts
03:40:00 0.35 0.61 0.61 0.00 60.00
03:45:00 0.03 0.06 0.06 0.00 0.00
03:50:00 0.02 0.03 0.02 0.00 0.00
03:55:00 0.02 0.02 0.02 0.00 0.00
bash-3.00# sar -u
SunOS XXXX 5.10 Generic_118833-20 sun4u 09/02/2006
00:00:00 %usr %sys %wio %idle
[...]
02:00:00 0 1 0 99
02:05:00 0 1 0 99
02:10:00 0 1 0 99
02:15:01 0 1 0 99
02:20:00 0 15 0 85
02:25:00 0 34 0 66
02:30:00 0 20 0 80
02:35:00 0 22 0 78
02:40:01 0 45 0 55
02:45:02 0 61 0 38
03:39:28 unix restarts
03:40:00 5 10 0 84
03:45:00 1 1 0 98
03:50:00 0 0 0 100
bash-3.00# sar -q
SunOS xxx 5.10 Generic_118833-20 sun4u 09/02/2006
00:00:00 runq-sz %runocc swpq-sz %swpocc
[...]
02:00:00 0.0 0 0.0 0
02:05:00 1.0 0 0.0 0
02:10:00 0.0 0 0.0 0
02:15:01 0.0 0 0.0 0
02:20:00 1.1 5 0.0 0
02:25:00 1.4 12 0.0 0
02:30:00 2.1 6 0.0 0
02:35:00 3.4 9 0.0 0
02:40:01 2.8 25 0.0 0
02:45:02 4.0 44 116.6 12
03:39:28 unix restarts
03:40:00 1.0 3 0.0 0
03:45:00 0.0 0 0.0 0
03:50:00 0.0 0 0.0 0
Crashdump could be provided off-the list and not for public eyes.
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
