On Dec 12, 2006, at 4:24 PM, Gleb Natapov wrote:
On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote:
Well I have no luck in finding a way to up the amount the system
will
allow GM to use. What is a recommended solution? Is this even a
problem in most cases? Like am i encountering a corner case?
upping the limit was not what i'm suggesting as a fix, just
pointing out
that it is kind of low and even with a fully working ompi or mpich-
gm. ompi
should still work, even if the IOMMU limit is low.
Since you are running 1 thread per CPU (== 2 total), it is
possible (likely)
that the 1st thread is grabbing all the available registerable
memory,
leaving not even enough for the second thread to even start. I
recommend
you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
equivalent setting is used in MPICH-GM in similar situations. Set
this to
about 180 MB and run with that.
Gleb - I assume that when registration needs exceed
"mpool_rdma_rcache_size_limit", that previously registered memory is
unregistered much as virtual memory is swapped out?
If previously registered memory is in use than registration returns
error to upper layer and operation is retried late. Otherwise
unused memory
is unregistered. The code for mpool_rdma_rcache_size_limit is not on
trunk yet. It is on tmp branch /tmp/gleb-mpool, I don't know if /
tmp is
open to everyone. If not I can send the patch.
If you could mail me the patch ill try it out.
--
Gleb.
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
