Re: Memory pressure handling with iSCSI

2005-07-26 Thread Martin J. Bligh
> I don't think so. We're getting the wrong answer out of > calculate_zone_totalpages() which is an init-time thing. > > Maybe nr_free_zone_pages() is supposed to fix that up post-facto somehow, > but calculate_zone_totalpages() sure as heck shouldn't be putting 1568768 > into my ZONE_NORMAL's

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > This machine only has 4G of memory, so the platform code is overestimating > > the number of pages by 50%. Can you please check your dmesg, see if your > > system is also getting this wrong? > > > > On node 0 totalpages: 1572863 > DMA zone: 40

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 17:31 -0700, Andrew Morton wrote: > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > On Tue, 2005-07-26 at 16:07 -0700, Andrew Morton wrote: > > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > > > Here is the data with 5 ext2 filesystems. I also collected > >

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
"Martin J. Bligh" <[EMAIL PROTECTED]> wrote: > > > > It happens here, a bit. My machine goes up to 60% dirty when it should be > > clamping at 40%. > > > > The variable `total_pages' in page-writeback.c (from > > nr_free_pagecache_pages()) is too high. I trace it back to here: > > > > On node

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Martin J. Bligh
> It happens here, a bit. My machine goes up to 60% dirty when it should be > clamping at 40%. > > The variable `total_pages' in page-writeback.c (from > nr_free_pagecache_pages()) is too high. I trace it back to here: > > On node 0 totalpages: 1572864 > DMA zone: 4096 pages, LIFO batch:1 >

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > On Tue, 2005-07-26 at 16:07 -0700, Andrew Morton wrote: > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > Here is the data with 5 ext2 filesystems. I also collected /proc/meminfo > > > every 5 seconds. As you can see, we seem to dirty 6G

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 16:07 -0700, Andrew Morton wrote: > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > Here is the data with 5 ext2 filesystems. I also collected /proc/meminfo > > every 5 seconds. As you can see, we seem to dirty 6GB of data in 20 > > seconds of starting the test. I am not s

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > Here is the data with 5 ext2 filesystems. I also collected /proc/meminfo > every 5 seconds. As you can see, we seem to dirty 6GB of data in 20 > seconds of starting the test. I am not sure if its bad, since we have > lots of free memory.. It's bad.

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 15:10 -0700, Andrew Morton wrote: > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > On Tue, 2005-07-26 at 14:24 -0700, Andrew Morton wrote: > > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > > > ext2 is incredibly better. Machine is very responsive. > > > > >

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Adam Litke
On Tue, 2005-07-26 at 16:33, Martin J. Bligh wrote: > >> > After KS & OLS discussions about memory pressure, I wanted to re-do > >> > iSCSI testing with "dd"s to see if we are throttling writes. > >> > >> Could you also try with shared writable mmap, to see if that > >> works ok or triggers a de

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > On Tue, 2005-07-26 at 14:24 -0700, Andrew Morton wrote: > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > ext2 is incredibly better. Machine is very responsive. > > > > > > > OK. Please, always monitor and send /proc/meminfo. I assume t

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 14:24 -0700, Andrew Morton wrote: > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > ext2 is incredibly better. Machine is very responsive. > > > > OK. Please, always monitor and send /proc/meminfo. I assume that the > dirty-memory clamping is working OK with ext2 and

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Martin J. Bligh
>> > After KS & OLS discussions about memory pressure, I wanted to re-do >> > iSCSI testing with "dd"s to see if we are throttling writes. >> >> Could you also try with shared writable mmap, to see if that >> works ok or triggers a deadlock ? > > > I can, but lets finish addressing one issue a

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > ext2 is incredibly better. Machine is very responsive. > OK. Please, always monitor and send /proc/meminfo. I assume that the dirty-memory clamping is working OK with ext2 and that perhaps it'll work OK with ext3/data=writeback. All very odd. I

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > You probably covered this, but just to make sure, if you're on a > > pentium4 machine, I usually boot w/ "idle=poll" to see proper idle > > reporting because otherwise the chip will throttle itself back and > > idle time will be skewed -- at least o

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 12:12 -0700, Andrew Morton wrote: > Andrew Morton <[EMAIL PROTECTED]> wrote: > > > > Can you please reduce the number of filesystems, see if that reduces the > > dirty levels? > > Also, it's conceivable that ext3 is implicated here, so it might be saner > to perform initial

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 16:59 -0400, Rik van Riel wrote: > On Tue, 26 Jul 2005, Badari Pulavarty wrote: > > > After KS & OLS discussions about memory pressure, I wanted to re-do > > iSCSI testing with "dd"s to see if we are throttling writes. > > Could you also try with shared writable mmap, to s

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Rik van Riel <[EMAIL PROTECTED]> wrote: > > On Tue, 26 Jul 2005, Badari Pulavarty wrote: > > > After KS & OLS discussions about memory pressure, I wanted to re-do > > iSCSI testing with "dd"s to see if we are throttling writes. > > Could you also try with shared writable mmap, to see if that >

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Rik van Riel
On Tue, 26 Jul 2005, Badari Pulavarty wrote: > After KS & OLS discussions about memory pressure, I wanted to re-do > iSCSI testing with "dd"s to see if we are throttling writes. Could you also try with shared writable mmap, to see if that works ok or triggers a deadlock ? -- The Theory of Esc

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 15:31 -0400, Sonny Rao wrote: > On Tue, Jul 26, 2005 at 11:39:11AM -0700, Badari Pulavarty wrote: > > On Tue, 2005-07-26 at 11:11 -0700, Andrew Morton wrote: > > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > > > After KS & OLS discussions about memory pressure, I w

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 12:12 -0700, Andrew Morton wrote: > Andrew Morton <[EMAIL PROTECTED]> wrote: > > > > Can you please reduce the number of filesystems, see if that reduces the > > dirty levels? > > Also, it's conceivable that ext3 is implicated here, so it might be saner > to perform initial

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Sonny Rao
On Tue, Jul 26, 2005 at 11:39:11AM -0700, Badari Pulavarty wrote: > On Tue, 2005-07-26 at 11:11 -0700, Andrew Morton wrote: > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > After KS & OLS discussions about memory pressure, I wanted to re-do > > > iSCSI testing with "dd"s to see if we ar

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Andrew Morton <[EMAIL PROTECTED]> wrote: > > Can you please reduce the number of filesystems, see if that reduces the > dirty levels? Also, it's conceivable that ext3 is implicated here, so it might be saner to perform initial investigation on ext2. (when kjournald writes back a page via its buf

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > On Tue, 2005-07-26 at 11:11 -0700, Andrew Morton wrote: > > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > > > After KS & OLS discussions about memory pressure, I wanted to re-do > > > iSCSI testing with "dd"s to see if we are throttling writes.

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Badari Pulavarty
On Tue, 2005-07-26 at 11:11 -0700, Andrew Morton wrote: > Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > > > After KS & OLS discussions about memory pressure, I wanted to re-do > > iSCSI testing with "dd"s to see if we are throttling writes. > > > > I created 50 10-GB ext3 filesystems on iSCS

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Andrew Morton
Badari Pulavarty <[EMAIL PROTECTED]> wrote: > > After KS & OLS discussions about memory pressure, I wanted to re-do > iSCSI testing with "dd"s to see if we are throttling writes. > > I created 50 10-GB ext3 filesystems on iSCSI luns. Test is simple > 50 dds (one per filesystem). System seems

Re: Memory pressure handling with iSCSI

2005-07-26 Thread Roland Dreier
Badari> I created 50 10-GB ext3 filesystems on iSCSI luns. Test is Badari> simple 50 dds (one per filesystem). System seems to Badari> throttle memory properly and making progress. (Machine Badari> doesn't respond very well for anything else, but my vmstat Badari> keeps running