On Mon, Dec 11, 2006 at 02:52:40PM -0500, Brock Palen wrote: > On Dec 11, 2006, at 2:45 PM, Reese Faucette wrote: > > >> Also I have no idea what the memory window question is, i will > >> look it up on google. > >> > >> aon075:~ root# dmesg | grep GM > >> GM: gm_register_memory will be able to lock 96000 pages (375 MBytes) > > > > This just answered it - there is 375MB available for GM to > > register, which > > is the IOMMU window size available to the GM driver. This is quite > > small, > > and it would be very helpful if this could be increased. helpful > > == much > > better performance for your jobs. > Always a plus > > > > > It's possible that OMPI is not managing registered space well, as is > > required when the aggregate registered memory needed by a job is > > larger than > > the memory available to be registered. I do some research locally > > into > > OMPIs management of registered memory when using GM. If it comes > > to it, > > would you be willing to run an OMPI with some debug statements in > > it for me? > > thanks, > Sure i can run them. I added to OMPI possibility to limit the amount of registered memory. There is a new parameter mpool_rdma_rcache_size_limit that controls how much memory can be pinned at once. But this work is not yet on the trunk. If you can checkout /tmp/gleb-mpool branch and test it this would be great.
> > > -reese > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Gleb.
