Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-17 Thread Mark Martinec
On 07/08/2018 15:58, Mark Martinec wrote: Collected, here it is:   https://www.ijs.si/usr/mark/tmp/dtrace-cmd.out.bz2 2018-08-14 11:18, Andriy Gapon wrote: I see one memory leak, not sure if it's the only one. It looks like vdev_geom_read_config() leaks all parsed vdev nvlist-s but the last

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-14 Thread Andriy Gapon
On 07/08/2018 15:58, Mark Martinec wrote: > Collected, here it is: > >   https://www.ijs.si/usr/mark/tmp/dtrace-cmd.out.bz2 I see one memory leak, not sure if it's the only one. It looks like vdev_geom_read_config() leaks all parsed vdev nvlist-s but the last. The problems seems to come from r3

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-13 Thread Mark Martinec
2018-08-13 21:48, Volodymyr Kostyrko wrote: I've been in the same situation. ZFS, only pool, no ZFS errors. I think the problem is rather between swapping and ZFS ARC. This host has different load, sometimes it needs more active memory, somtimes less... This means that active zone can expand and

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-13 Thread Volodymyr Kostyrko
23.07.18 18:12, Mark Martinec wrote: After upgrading an older AMD host from FreeBSD 10.3 to 11.1-RELEASE-p11 (amd64), ZFS is gradually eating up all memory, so that it crashes every few days when the memory is completely exhausted (after swapping heavily for a couple of hours). I've been in the

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-07 Thread Mark Martinec
On Sat, Aug 04, 2018 at 08:38:04PM +0200, Mark Martinec wrote: 2018-08-04 19:01, Mark Johnston wrote: > I think running "zpool list" is adding a lot of noise to the output. > Could you retry without doing that? No, like I said previously, the "zpool list" (with one defunct zfs pool) *is* the sole

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Johnston
On Sat, Aug 04, 2018 at 08:38:04PM +0200, Mark Martinec wrote: > 2018-08-04 19:01, Mark Johnston wrote: > > I think running "zpool list" is adding a lot of noise to the output. > > Could you retry without doing that? > > No, like I said previously, the "zpool list" (with one defunct > zfs pool) *i

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Martinec
2018-08-04 19:01, Mark Johnston wrote: I think running "zpool list" is adding a lot of noise to the output. Could you retry without doing that? No, like I said previously, the "zpool list" (with one defunct zfs pool) *is* the sole culprit of the zfs memory leak. With each invocation of "zpool l

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Johnston
On Fri, Aug 03, 2018 at 09:11:42PM +0200, Mark Martinec wrote: > More attempts at tracking this down. The suggested dtrace command does > usually abort with: > >Assertion failed: (buf->dtbd_timestamp >= first_timestamp), > file > /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-03 Thread Mark Martinec
More attempts at tracking this down. The suggested dtrace command does usually abort with: Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c, line 3330. but with some luck soon after each machine reboo

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-01 Thread Mark Martinec
On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote: I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE and the situation has not improved. Also turned off all services. ZFS is still leaking memory about 30 MB per hour, until the host runs out of memory and swap space a

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Nagy László Zsolt
I have experienced a very similar thing. After upgrading my machine from 11.1-R to 11.2-R, the swap space is filled up to about 66% in about every 2 days. First I tought that it was PostgreSQL, and lowered the shared_buffers setting, but it only postponed the problem for another day. The only thin

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Shane Ambler
On 01/08/2018 07:24, Mark Martinec wrote: > I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE > and the situation has not improved. Also turned off all services. > ZFS is still leaking memory about 30 MB per hour, until the host > runs out of memory and swap space and crashes, unl

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Mark Johnston
On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote: > I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE > and the situation has not improved. Also turned off all services. > ZFS is still leaking memory about 30 MB per hour, until the host > runs out of memory and swap s

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Mark Martinec
I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE and the situation has not improved. Also turned off all services. ZFS is still leaking memory about 30 MB per hour, until the host runs out of memory and swap space and crashes, unless I reboot it first every four days. Any advis