Re: Still getting kmem exhausted panic

2010-09-29 Thread Andriy Gapon
on 29/09/2010 03:38 Artem Belevich said the following: > On Tue, Sep 28, 2010 at 3:22 PM, Andriy Gapon wrote: >> BTW, have you seen my posts about UMA and ZFS on hackers@ ? >> I found it advantageous to use UMA for ZFS I/O buffers, but only after >> reducing >> size of per-CPU caches for the zone

Re: Still getting kmem exhausted panic

2010-09-28 Thread Artem Belevich
On Tue, Sep 28, 2010 at 3:22 PM, Andriy Gapon wrote: > BTW, have you seen my posts about UMA and ZFS on hackers@ ? > I found it advantageous to use UMA for ZFS I/O buffers, but only after > reducing > size of per-CPU caches for the zones with large-sized items. > I further modified the code in my

Still getting kmem exhausted panic

2010-09-28 Thread Leroy van Logchem
>> Thanks for the clarification. I just wish I knew how vm.kmem_size_scale >> fit into the picture (meaning what it does, etc.). The sysctl >> description isn't very helpful. Again, my lack of VM knowledge... >> >Roughly, vm.kmem_size would get set to divided by >vm.kmem_size_scale. http://lis

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 29/09/2010 01:01 Ben Kelly said the following: > Thanks. Yea, there is a lot of aggressive tuning there. In particular, the > slow growth algorithm is somewhat dubious. What I found, though, was that > the fragmentation jumped whenever the arc was reduced in size, so it was an > attempt to ma

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 5:30 PM, Andriy Gapon wrote: << snipped lots of good info here... probably won't have time to look at it in detail until the weekend >> >> there seems to be a layering violation in that the buffer cache signals >> directly to the upper page daemon layer to trigger page recla

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 21:40 Ben Kelly said the following: > > On Sep 28, 2010, at 1:17 PM, Andriy Gapon wrote: > >> on 28/09/2010 19:46 Ben Kelly said the following: >>> Hmm. My server is currently idle with no I/O happening: >>> >>> kstat.zfs.misc.arcstats.c: 25165824 kstat.zfs.misc.arcstats.c_max: >>

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 1:17 PM, Andriy Gapon wrote: > on 28/09/2010 19:46 Ben Kelly said the following: >> Hmm. My server is currently idle with no I/O happening: >> >> kstat.zfs.misc.arcstats.c: 25165824 >> kstat.zfs.misc.arcstats.c_max: 46137344 >> kstat.zfs.misc.arcstats.size: 91863156 >> >

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 20:17 Andriy Gapon said the following: > on 28/09/2010 19:46 Ben Kelly said the following: >> If what you say is true, this shouldn't happen, should it? This system is >> an i386 machine with kmem max at 800M and arc set to 40M. This is running >> head from April 6, 2010, so it is

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 19:46 Ben Kelly said the following: > Hmm. My server is currently idle with no I/O happening: > > kstat.zfs.misc.arcstats.c: 25165824 > kstat.zfs.misc.arcstats.c_max: 46137344 > kstat.zfs.misc.arcstats.size: 91863156 > > If what you say is true, this shouldn't happen, should

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 12:30 PM, Andriy Gapon wrote: > on 28/09/2010 18:50 Ben Kelly said the following: >> >> On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: >>> Well, no time for me to dig through all that history. arc_max should be a >>> hard limit and it is now. If it ever wasn't then it was a

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 18:50 Ben Kelly said the following: > > On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: >> Well, no time for me to dig through all that history. arc_max should be a >> hard limit and it is now. If it ever wasn't then it was a bug. > > I believe the size of the arc could exceed the

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: > Well, no time for me to dig through all that history. > arc_max should be a hard limit and it is now. If it ever wasn't then it was a > bug. I believe the size of the arc could exceed the limit if your working set was larger than arc_max. The

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:30 Willem Jan Withagen said the following: > So in my case (no other serious apps) with 12G phys mem: > > vm.kmem_size=17G > vfs.zfs.arc_max=11G > Should be good. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lis

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 16:25, Andriy Gapon wrote: > on 28/09/2010 17:09 Willem Jan Withagen said the following: >> On 28-9-2010 16:07, Andriy Gapon wrote: >>> on 28/09/2010 17:02 Willem Jan Withagen said the following: I do have (read) this document, but st

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:09 Willem Jan Withagen said the following: > On 28-9-2010 16:07, Andriy Gapon wrote: >> on 28/09/2010 17:02 Willem Jan Withagen said the following: >>> I do have (read) this document, but still that doesn't really give you >>> guidelines for tuning on FreeBSD. It is a fileserver wi

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 16:07, Andriy Gapon wrote: > on 28/09/2010 17:02 Willem Jan Withagen said the following: >> I do have (read) this document, but still that doesn't really give you >> guidelines for tuning on FreeBSD. It is a fileserver without any serious

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:02 Willem Jan Withagen said the following: > I do have (read) this document, but still that doesn't really give you > guidelines for tuning on FreeBSD. It is a fileserver without any serious > other apps. > I was using "auto-tuned", and that crashed my box. That is what started > t

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 15:46, Andriy Gapon wrote: > on 28/09/2010 16:25 Willem Jan Withagen said the following: >> Well advises seem to vary, and the latest I understood was that >> 8.1-stable did not need any tuning. (The other system with a much >> older kern

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:25 Willem Jan Withagen said the following: > Well advises seem to vary, and the latest I understood was that > 8.1-stable did not need any tuning. (The other system with a much older > kernel is tuned as to what most here are suggesting) > And I was shure led to believe that even s

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:23 Jeremy Chadwick said the following: > On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: >> I believe that "the trick" is to set vm.kmem_size high enough, eitehr using >> this >> tunable or vm.kmem_size_scale. > > Thanks for the clarification. I just wish I knew ho

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:23 Jeremy Chadwick said the following: > On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: >> on 28/09/2010 14:50 Jeremy Chadwick said the following: >>> I believe the trick -- Andriy, please correct me if I'm wrong -- is the >> >> Wouldn't hurt to CC me, so that I coul

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 13:50, Jeremy Chadwick wrote: > On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote: >> This is with stable as of yesterday,but with an un-tunned ZFS box I >> was still able to generate a kmem exhausted panic. >> Hard panic

Re: Still getting kmem exhausted panic

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: > on 28/09/2010 14:50 Jeremy Chadwick said the following: > > I believe the trick -- Andriy, please correct me if I'm wrong -- is the > > Wouldn't hurt to CC me, so that I could do it :-) > > > tuning of vfs.zfs.arc_max, which is now a

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 14:50 Jeremy Chadwick said the following: > I believe the trick -- Andriy, please correct me if I'm wrong -- is the Wouldn't hurt to CC me, so that I could do it :-) > tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high > watermark". Not sure what you mean here

Re: Still getting kmem exhausted panic

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote: > This is with stable as of yesterday,but with an un-tunned ZFS box I > was still able to generate a kmem exhausted panic. > Hard panic, just 3 lines. > > The box contains 12Gb memory, runs on a 6 core (with HT) xeon. > 6* 2T WD

Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
Hi, This is with stable as of yesterday,but with an un-tunned ZFS box I was still able to generate a kmem exhausted panic. Hard panic, just 3 lines. The box contains 12Gb memory, runs on a 6 core (with HT) xeon. 6* 2T WD black caviar in raidz2 with 2*512Mb mirrored log. The box died while rsy