I think Ashley still has the right general idea.
You need to see how much memory the OS is taking off the top. Then
see how much memory the application images consume (before using any
memory). Open MPI itself then takes up a bunch of memory for its own
internal buffering. Remember, too, that Open MPI will default to both
shared memory and OpenFabrics -- both of which have their own,
separate buffering.
You can disable shared memory, if you want, by specifying
mpirun --mca btl openib,self ...
("sm" is the shared memory btl, so by not specifying it, Open MPI
won't use it)
If you have recent Mellanox HCAs, you should probably be using OMPI's
XRC support, which will decrease OMPI's memory usage even further
(I'll let Mellanox comment on this further if they want).
Finally, there's a bunch of information on this FAQ page describing
how to tune Open MPI's OpenFabrics usage:
http://www.open-mpi.org/faq/?category=openfabrics
On Apr 23, 2009, at 1:35 PM, Viral Mehta wrote:
yes of course i m sure about wellness of providers.... and i m using
ofed-1.4.1-rc3
i m running 24 proc per node on 8 node cluster..... so as i showed
in calculation that i require 36G mem.....
i just need to know if my calculation has not some obvious flaw and/
or if i m missing anything about setting up system environment or
anything like that
On Thu, Apr 23, 2009 at 10:36 PM, gossips J <polk...@gmail.com> wrote:
What is the NIC you use?
What OFED build?
Are you sure about wellness of provider lib/drivers..?
It is strange that you get out of mem in all to all tests... should
not happen on 32G system,..!!!
-polk.
On 4/23/09, viral....@gmail.com <viral....@gmail.com> wrote:
or any link which helps to understand system reuirement for certain
test scenario ..
On Apr 23, 2009 12:42pm, viral....@gmail.com wrote:
> Hi
> Thanks for your response.
> However, I am running
> mpiexec .... -ppn 24 -n 192 /opt/IMB-MPI1 alltaoll -msglen /root/
temp
>
> And file /root/temp contains entry upto 65535 size only. That
means alltoall test will run upto 65K size only
>
> So, in that case I will require very less memory but then in that
case also test is running out-of-memory. Please help someone to
understand the scenario.
> Or do I need to switch to some algorithm or do I need to set some
other environment variables ? or anything like that ?
>
> On Apr 22, 2009 6:43pm, Ashley Pittman ash...@pittman.co.uk> wrote:
> > On Wed, 2009-04-22 at 12:40 +0530, vkm wrote:
> >
> >
> >
> > > The same amount of memory required for recvbuf. So at the
least each
> >
> > > node should have 36GB of memory.
> >
> > >
> >
> > > Am I calculating right ? Please correct.
> >
> >
> >
> > Your calculation looks correct, the conclusion is slightly wrong
> >
> > however. The Application buffers will consume 36Gb of memory,
the rest
> >
> > of the application, any comms buffers and the usual OS overhead
will be
> >
> > on top of this so putting only 36Gb of ram in your nodes will
still
> >
> > leave you short.
> >
> >
> >
> > Ashley,
> >
> >
> >
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Thanks,
Viral Mehta
--
Jeff Squyres
Cisco Systems
