On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote:
> > Well I have no luck in finding a way to up the amount the system will
> > allow GM to use. What is a recommended solution? Is this even a
> > problem in most cases? Like am i encountering a corner case?
>
> upping the limit was not what i'm suggesting as a fix, just pointing out
> that it is kind of low and even with a fully working ompi or mpich-gm. ompi
> should still work, even if the IOMMU limit is low.
>
> Since you are running 1 thread per CPU (== 2 total), it is possible (likely)
> that the 1st thread is grabbing all the available registerable memory,
> leaving not even enough for the second thread to even start. I recommend
> you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
> equivalent setting is used in MPICH-GM in similar situations. Set this to
> about 180 MB and run with that.
>
> Gleb - I assume that when registration needs exceed
> "mpool_rdma_rcache_size_limit", that previously registered memory is
> unregistered much as virtual memory is swapped out?
>
If previously registered memory is in use than registration returns
error to upper layer and operation is retried late. Otherwise unused memory
is unregistered. The code for mpool_rdma_rcache_size_limit is not on
trunk yet. It is on tmp branch /tmp/gleb-mpool, I don't know if /tmp is
open to everyone. If not I can send the patch.
--
Gleb.
