On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote:
I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE
and the situation has not improved. Also turned off all services.
ZFS is still leaking memory about 30 MB per hour, until the host
runs out of memory and swap space and crashes, unless I reboot it
first every four days.
Any advise before I try to get rid of that faulted disk with a pool
(or downgrade to 10.3, which was stable) ?
2018-08-01 00:09, Mark Johnston wrote:
If you're able to use dtrace, it would be useful to try tracking
allocations with the solaris tag:
# dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] =
count()} dtmalloc::solaris:free {@frees[stack(), args[3]] =
count();}'
Try letting that run for one minute, then kill it and paste the output.
Ideally the host will be as close to idle as possible while still
demonstrating the leak.
Good and bad news:
The suggested dtrace command bails out:
# dtrace -n 'dtmalloc::solaris:malloc {@allocs[stack(), args[3]] =
count()} dtmalloc::solaris:free {@frees[stack(), args[3]] = count();}'
dtrace: description 'dtmalloc::solaris:malloc ' matched 2 probes
Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file
/usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c,
line 3330.
Abort trap
But I did get one step further, localizing the culprit.
I realized that the "solaris" malloc count goes up in sync with
the 'telegraf' monitoring service polls, which also has a ZFS plugin
which monitors the zfs pool and ARC. This plugin runs 'zpool list -Hp'
periodically.
So after stopping telegraf (and other remaining services),
the 'vmstat -m' shows that InUse count for "solaris" goes up by 552
every time that I run "zpool list -Hp" :
# (while true; do zpool list -Hp >/dev/null; vmstat -m | \
fgrep solaris; sleep 1; done) | awk '{print $2-a; a=$2}'
6664427
541
552
552
552
552
552
552
552
552
556
548
552
552
552
552
552
552
552
552
552
# zpool list -Hp
floki 68719476736 37354102272 31365374464 - -
49% 54 1.00x ONLINE -
stuff - - - - - - - -
UNAVAIL -
Mark
_______________________________________________
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
