Thanks again.
With Blessings, always,
Jerry Mersel
[cid:image003.png@01CF80E7.62B7D810]
System Administrator
IT Infrastructure Branch | Division of Information Systems
Weizmann Institute of Science
Rehovot 76100, Israel
Tel: +972-8-9342363
“allow our heart, the heart of each and every one of us, to to see the good
qualities of our friends and not their shortcomings...”
--Reb Elimelech of Lizhensk
"We learn something by doing it. There is no other way."
—John Holt
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Thursday, November 20, 2014 5:42 PM
To: Open MPI Users
Subject: Re: [OMPI users] job running out of memory
It wouldn’t be maffinity - that just tells the OS on a single node to ensure
that the memory is local to the process.
If you have a managed environment (i.e., there is a scheduler running that
assigns nodes to jobs), then you would need to configure the scheduler to
support that option (assuming it does, you usually have to tell it to turn that
option on), and the user would have to specify that they want “large memory”
nodes.
If it is an unmanaged environment where you are specifying nodes using a
hostfile, then you could provide a hostfile that contains “large memory” nodes.
If you only have a limited number of large memory nodes, you know which rank is
the one in question, and you don’t want to use large memory nodes for the
entire job, then things get a little more complicated. OMPI doesn’t have an
automatic mapper that handles this use-case at the moment, so you would have to
create a file that tells us where to put the various ranks, or do it via the
cmd line:
* command line: use the same trick I gate you earlier for valgrind
mpirun -n (M-1) -host ^large myapp : mpirun -n 1 -host large myapp : mpirun -n
(N-M) -host ^large myapp
* rankfile - stipulate where to run each rank (see "man mpirun” for syntax)
* seq - provide a list of node names and we’ll put each rank in order on the
nodes in the list.
If you don’t know which process(es) are memory intensive, then there really
isn’t anything OMPI can do - there is no way for us to know in advance which
ranks will take a lot of memory.
HTH
Ralph
On Nov 19, 2014, at 11:51 PM, Jerry Mersel
<jerry.mer...@weizmann.ac.il<mailto:jerry.mer...@weizmann.ac.il>> wrote:
Since there is no way to share memory across nodes is there a way to tell
openmpi to run memory intensive
Processes on nodes with a lot of memory?
I think that it is called “maffinity”, but am not sure how to use it.
With Blessings, always,
Jerry Mersel
<image001.png>
System Administrator
IT Infrastructure Branch | Division of Information Systems
Weizmann Institute of Science
Rehovot 76100, Israel
Tel: +972-8-9342363
“allow our heart, the heart of each and every one of us, to to see the good
qualities of our friends and not their shortcomings...”
--Reb Elimelech of Lizhensk
"We learn something by doing it. There is no other way."
—John Holt
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Tuesday, November 18, 2014 5:56 PM
To: Open MPI Users
Subject: Re: [OMPI users] job running out of memory
Unfortunately, there is no way to share memory across nodes. Running out of
memory as you describe can be due to several factors, including most typically:
* a memory leak in the application, or the application simply growing too big
for the environment
* one rank running slow, causing it to build a backlog of messages that
eventually consumes too much memory
What you can do:
* add --display-map to your mpirun cmd line to see what MPI ranks are running
on each node
* run "top" on each node to see which process is getting too large. Generally,
you'll see the pid for the processes run in order from the lowest local rank to
the highest since that is the order in which they are spawned, so you can
figure out which rank is causing the problem.
* run valgrind on that rank. This isn't as hard as it might sound, though the
cmd line is a tad tricky. If rank M (out of a total job of N ranks) is the one
with the memory problem, then you would do this:
mpirun -n (M-1) myapp : mpirun -n 1 valgrind myapp : mpirun -n (N-M) myapp
Obviously, you can add whatever options you want to the various pieces of the
cmd line
This will affect the timing, so a race condition might be hidden - but it is a
start at tracking it down.
If you find that this isn't a leak, but rather a legitimate behavior, then you
can try using the mapping and ranking options to redistribute the processes -
maybe oversubscribe some of the nodes or increase the size of the allocation so
the memory hog can run alone.
HTH
Ralph
On Tue, Nov 18, 2014 at 7:10 AM, Jerry Mersel
<jerry.mer...@weizmann.ac.il<mailto:jerry.mer...@weizmann.ac.il>> wrote:
Hi all:
I am running openmpi 1.6.5 and a job which is memory intensive.
The job runs on 7 hosts using 16 core on each. On one of the hosts the memory
is exhausted so the kernel starts to
Kill the processes.
It could be that there is plenty of free memory on one of the other hosts.
Is there a way for openmpi to use the memory on one of the other hosts when
the memory on one host is gone.
If yes please tell me how.
Thank you.
With Blessings, always,
Jerry Mersel
<image001.png>
System Administrator
IT Infrastructure Branch | Division of Information Systems
Weizmann Institute of Science
Rehovot 76100, Israel
Tel: +972-8-9342363<tel:%2B972-8-9342363>
“allow our heart, the heart of each and every one of us, to to see the good
qualities of our friends and not their shortcomings...”
--Reb Elimelech of Lizhensk
"We learn something by doing it. There is no other way."
—John Holt
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/11/25839.php
_______________________________________________
users mailing list
us...@open-mpi.org<mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2014/11/25846.php
