Re: [OMPI users] mpool_gm_module error

2006-12-12 Thread Brock Palen
On Dec 12, 2006, at 4:24 PM, Gleb Natapov wrote: On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote: Well I have no luck in finding a way to up the amount the system will allow GM to use. What is a recommended solution? Is this even a problem in most cases? Like am i encounterin

Re: [OMPI users] mpool_gm_module error

2006-12-12 Thread Gleb Natapov
On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote: > > Well I have no luck in finding a way to up the amount the system will > > allow GM to use. What is a recommended solution? Is this even a > > problem in most cases? Like am i encountering a corner case? > > upping the limit was

Re: [OMPI users] mpool_gm_module error

2006-12-12 Thread Reese Faucette
Well I have no luck in finding a way to up the amount the system will allow GM to use. What is a recommended solution? Is this even a problem in most cases? Like am i encountering a corner case? upping the limit was not what i'm suggesting as a fix, just pointing out that it is kind of low an

Re: [OMPI users] mpool_gm_module error

2006-12-12 Thread Brock Palen
On Dec 11, 2006, at 4:04 PM, Reese Faucette wrote: GM: gm_register_memory will be able to lock XXX pages (YYY MBytes) Is there a way to tell GM to pull more memory from the system? GM reserves all IOMMU space that the OS is willing to give it, so what is needed is a way to tell the OS and/o

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Reese Faucette
GM: gm_register_memory will be able to lock XXX pages (YYY MBytes) Is there a way to tell GM to pull more memory from the system? GM reserves all IOMMU space that the OS is willing to give it, so what is needed is a way to tell the OS and/or machine to allow a bigger chunk of IOMMU space to b

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Brock Palen
On Dec 11, 2006, at 2:20 PM, Reese Faucette wrote: I have tried moving around machines that the run is done on to the same result in multiple places. The error is: [aon049.engin.umich.edu:21866] [mpool_gm_module.c:100] error(8) registering gm memory This is on a PPC-based OSX system? How man

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Gleb Natapov
On Mon, Dec 11, 2006 at 02:52:40PM -0500, Brock Palen wrote: > On Dec 11, 2006, at 2:45 PM, Reese Faucette wrote: > > >> Also I have no idea what the memory window question is, i will > >> look it up on google. > >> > >> aon075:~ root# dmesg | grep GM > >> GM: gm_register_memory will be able to lo

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Brock Palen
On Dec 11, 2006, at 2:45 PM, Reese Faucette wrote: Also I have no idea what the memory window question is, i will look it up on google. aon075:~ root# dmesg | grep GM GM: gm_register_memory will be able to lock 96000 pages (375 MBytes) This just answered it - there is 375MB available for GM t

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Reese Faucette
Also I have no idea what the memory window question is, i will look it up on google. aon075:~ root# dmesg | grep GM GM: gm_register_memory will be able to lock 96000 pages (375 MBytes) This just answered it - there is 375MB available for GM to register, which is the IOMMU window size available

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Brock Palen
On Dec 11, 2006, at 2:34 PM, Brock Palen wrote: Yes it is a PPC based system. The machines are duel G5 with 1 gig of ram. I am only running 1 thread per cpu, (not over allocating). It is not a maximized run, when running i see 500MB free on the nodes. Each thread uses ~110MB. Sorry to edit m

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Brock Palen
Yes it is a PPC based system. The machines are duel G5 with 1 gig of ram. I am only running 1 thread per cpu, (not over allocating). It is not a maximized run, when running i see 500MB free on the nodes. Each thread uses ~110MB. I could not answer wether or not OSX and PPC 970FX have a

Re: [OMPI users] mpool_gm_module error

2006-12-11 Thread Reese Faucette
I have tried moving around machines that the run is done on to the same result in multiple places. The error is: [aon049.engin.umich.edu:21866] [mpool_gm_module.c:100] error(8) registering gm memory This is on a PPC-based OSX system? How many MPI processes per node are you starting? And I as