Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)
Just to close the loop on this: I added the binding info to --display-map On Mar 28, 2014, at 6:58 AM, Jeff Squyres (jsquyres) wrote: > Good information; thanks. > > The short reason for this change in behavior of the affinity options is that > when we first created affinity (wy back in 1.0 days, no one really cared > about it much, and so we just did a first attempt). Gradually over time, > affinity has become much more important. As such, we have learned much from > what our users want and how they want to use affinity. That has caused a few > changes in approaches to how we do affinity -- and because our understanding > has grown, sometime it means that the changes we've made have been > revolutionary (vs. evolutionary), meaning that CLI options change, behaviors > change, etc. > > Sorry about that -- it really reflects how the whole HPC community is > evolving its attitude towards affinity over time. > > BTW, you should be aware that Open MPI v1.8 -- i.e., the next stable series > -- is scheduled to be released on Monday. There's additional changes with > regards to affinity in 1.8 (compared to the v1.6 series); much of what has > been discussed on this thread has been in the context of v1.7.x (which is > being renamed to 1.8 on Monday, per our "feature series eventually turns into > stable series" versioning philosophy). > > > > On Mar 28, 2014, at 9:47 AM, "Sasso, John (GE Power & Water, Non-GE)" > wrote: > >> Thanks again! I tried --display-devel-map and I think it provides a bit too >> much info for our needs. However, it is nice to know. >> >> BTW, some interesting behavior in using "--report-bindings --bind-to-core" >> vs "--display-map". >> >> * If I use "--report-bindings --bind-to-core" but the MPI tasks on a host >> fail to start up, then nothing is reported. For example, I had a problem >> where a job started across 4 hosts but the hosts could not communicate with >> one another via TCP/IP. >> >> * If I use "--display-map" then the mapping is shown, even in the failure >> case I mentioned in the last bullet. >> >> * What is nice about "--report-bindings --bind-to-core" over "--display-map" >> is that it will report the binding of each rank to CPU, whereas the latter >> will show you what ranks are running on a given host. For our needs, this >> may be sufficient, tho it would be nice to have the CPU bindings shown as >> well >> >> * If using "--report-bindings --bind-to-core" with OpenMPI 1.4.1 then the >> bindings on just the head node are shown. In 1.6.1, full bindings across >> all hosts are shown. (I'd have to read release notes on this...) >> >> --john >> >> >> -Original Message- >> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain >> Sent: Thursday, March 27, 2014 7:01 PM >> To: Open MPI Users >> Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages) >> >> Oooh...it's Jeff's fault! >> >> Fwiw you can get even more detailed mapping info with --display-devel-map >> >> Sent from my iPhone >> >>> On Mar 27, 2014, at 2:58 PM, "Jeff Squyres (jsquyres)" >>> wrote: >>> On Mar 27, 2014, at 4:06 PM, "Sasso, John (GE Power & Water, Non-GE)" wrote: Yes, I noticed that I could not find --display-map in any of the man pages. Intentional? >>> >>> Oops; nope. I'll ask Ralph to add it... >>> >>> -- >>> Jeff Squyres >>> jsquy...@cisco.com >>> For corporate legal information go to: >>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
[OMPI users] MPI_Barrier hangs on second attempt but only when multiple hosts used.
I have been using MPI for many many years so I have very well debugged mpi
tests. I am
having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions though
with getting the
MPI_Barrier calls to work. It works fine when I run all processes on one
machine but when
I run with two or more hosts the second call to MPI_Barrier always hangs.
Not the first one,
but always the second one. I looked at FAQ's and such but found nothing
except for a comment
that MPI_Barrier problems were often problems with fire walls. Also
mentioned as a problem
was not having the same version of mpi on both machines. I turned
firewalls off and removed
and reinstalled the same version on both hosts but I still see the same
thing. I then installed
lam mpi on two of my machines and that works fine. I can call the
MPI_Barrier function when run on
one of two machines by itself many times with no hangs. Only hangs if two
or more hosts are involved.
These runs are all being done on CentOS release 6.4. Here is test program
I used.
main (argc, argv)
int argc;
char **argv;
{
char message[20];
char hoster[256];
char nameis[256];
int fd, i, j, jnp, iret, myrank, np, ranker, recker;
MPI_Comm comm;
MPI_Status status;
MPI_Init( &argc, &argv );
MPI_Comm_rank( MPI_COMM_WORLD, &myrank);
MPI_Comm_size( MPI_COMM_WORLD, &np);
gethostname(hoster,256);
printf(" In rank %d and host= %s Do Barrier call
1.\n",myrank,hoster);
MPI_Barrier(MPI_COMM_WORLD);
printf(" In rank %d and host= %s Do Barrier call
2.\n",myrank,hoster);
MPI_Barrier(MPI_COMM_WORLD);
printf(" In rank %d and host= %s Do Barrier call
3.\n",myrank,hoster);
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
exit(0);
}
Here are three runs of test program. First with two processes on one
host, then with
two processes on another host, and finally with one process on each of two
hosts. The
first two runs are fine but the last run hangs on the second MPI_Barrier.
[root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out
In rank 0 and host= centos Do Barrier call 1.
In rank 1 and host= centos Do Barrier call 1.
In rank 1 and host= centos Do Barrier call 2.
In rank 1 and host= centos Do Barrier call 3.
In rank 0 and host= centos Do Barrier call 2.
In rank 0 and host= centos Do Barrier call 3.
[root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out
/root/.bashrc: line 14: unalias: ls: not found
In rank 0 and host= RAID Do Barrier call 1.
In rank 0 and host= RAID Do Barrier call 2.
In rank 0 and host= RAID Do Barrier call 3.
In rank 1 and host= RAID Do Barrier call 1.
In rank 1 and host= RAID Do Barrier call 2.
In rank 1 and host= RAID Do Barrier call 3.
[root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID a.out
/root/.bashrc: line 14: unalias: ls: not found
In rank 0 and host= centos Do Barrier call 1.
In rank 0 and host= centos Do Barrier call 2.
In rank 1 and host= RAID Do Barrier call 1.
In rank 1 and host= RAID Do Barrier call 2.
Since it is such a simple test and problem and such a widely used MPI
function, it must obviously
be an installation or configuration problem. A pstack for each of the
hung MPI_Barrier processes
on the two machines shows this:
[root@centos ~]# pstack 31666
#0 0x003baf0e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x7f5de06125eb in epoll_dispatch () from /usr/local/lib/libmpi.so.1
#2 0x7f5de061475a in opal_event_base_loop () from
/usr/local/lib/libmpi.so.1
#3 0x7f5de0639229 in opal_progress () from /usr/local/lib/libmpi.so.1
#4 0x7f5de0586f75 in ompi_request_default_wait_all () from
/usr/local/lib/libmpi.so.1
#5 0x7f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from
/usr/local/lib/openmpi/mca_coll_tuned.so
#6 0x7f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () from
/usr/local/lib/openmpi/mca_coll_tuned.so
#7 0x7f5de05941c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1
#8 0x00400a43 in main ()
[root@RAID openmpi-1.6.5]# pstack 22167
#0 0x0030302e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x7f7ee46885eb in epoll_dispatch () from /usr/local/lib/libmpi.so.1
#2 0x7f7ee468a75a in opal_event_base_loop () from
/usr/local/lib/libmpi.so.1
#3 0x7f7ee46af229 in opal_progress () from /usr/local/lib/libmpi.so.1
#4 0x7f7ee45fcf75 in ompi_request_default_wait_all () from
/usr/local/lib/libmpi.so.1
#5 0x7f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from
/usr/local/lib/openmpi/mca_coll_tuned.so
#6 0x7f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () from
/usr/local/lib/openmpi/mca_coll_tuned.so
#7 0x7f7ee460a1c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1
#8 0x00400a43 in main ()
Which looks exactly the same on each machine. Any thoughts or ideas would
be greatly appreciated as
I am stuck.
Clay Kirkland
