-mca btl_tcp_if_include 192.168.0.0/16 -mca oob_tcp_if_include 192.168.0.0/16
should do the trick. Any MPI is going to have trouble with your arrangement - just need a little hint to help figure it out. On May 6, 2014, at 5:14 PM, Clay Kirkland <clay.kirkl...@versityinc.com> wrote: > Someone suggested using some network address if all machines are on same > subnet. > They are all on the same subnet, I think. I have no idea what to put for a > param there. > I tried the ethernet address but of course it couldn't be that simple. Here > are my ifconfig > outputs from a couple of machines: > > [root@RAID MPI]# ifconfig -a > eth0 Link encap:Ethernet HWaddr 00:25:90:73:2A:36 > inet addr:192.168.0.59 Bcast:192.168.0.255 Mask:255.255.255.0 > inet6 addr: fe80::225:90ff:fe73:2a36/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:17983 errors:0 dropped:0 overruns:0 frame:0 > TX packets:9952 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:26309771 (25.0 MiB) TX bytes:758940 (741.1 KiB) > Interrupt:16 Memory:fbde0000-fbe00000 > > eth1 Link encap:Ethernet HWaddr 00:25:90:73:2A:37 > inet6 addr: fe80::225:90ff:fe73:2a37/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:56 errors:0 dropped:0 overruns:0 frame:0 > TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:3924 (3.8 KiB) TX bytes:468 (468.0 b) > Interrupt:17 Memory:fbee0000-fbf00000 > > And from one that I can't get to work: > > [root@centos ~]# ifconfig -a > eth0 Link encap:Ethernet HWaddr 00:1E:4F:FB:30:34 > inet6 addr: fe80::21e:4fff:fefb:3034/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:45 errors:0 dropped:0 overruns:0 frame:0 > TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:2700 (2.6 KiB) TX bytes:468 (468.0 b) > Interrupt:21 Memory:fe9e0000-fea00000 > > eth1 Link encap:Ethernet HWaddr 00:14:D1:22:8E:50 > inet addr:192.168.0.154 Bcast:192.168.0.255 Mask:255.255.255.0 > inet6 addr: fe80::214:d1ff:fe22:8e50/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:160 errors:0 dropped:0 overruns:0 frame:0 > TX packets:120 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:31053 (30.3 KiB) TX bytes:18897 (18.4 KiB) > Interrupt:16 Base address:0x2f00 > > > The centos machine is using eth1 and not eth0, therein lies the problem. > > I don't really need all this optimization of using multiple ethernet > adaptors to speed things > up. I am just using MPI to synchronize I/O tests. Can I go back to a > really old version > and avoid all this pain full debugging??? > > > > > On Tue, May 6, 2014 at 6:50 PM, <users-requ...@open-mpi.org> wrote: > Send users mailing list submissions to > us...@open-mpi.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.open-mpi.org/mailman/listinfo.cgi/users > or, via email, send a message with subject or body 'help' to > users-requ...@open-mpi.org > > You can reach the person managing the list at > users-ow...@open-mpi.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users digest..." > > > Today's Topics: > > 1. Re: users Digest, Vol 2881, Issue 1 (Clay Kirkland) > 2. Re: users Digest, Vol 2881, Issue 1 (Clay Kirkland) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 6 May 2014 18:28:59 -0500 > From: Clay Kirkland <clay.kirkl...@versityinc.com> > To: us...@open-mpi.org > Subject: Re: [OMPI users] users Digest, Vol 2881, Issue 1 > Message-ID: > <cajdnja90buhwu_ihssnna1a4p35+o96rrxk19xnhwo-nsd_...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > That last trick seems to work. I can get it to work once in a while with > those tcp options but it is > tricky as I have three machines and two of them use eth0 as primary network > interface and one > uses eth1. But by fiddling with network options and perhaps moving a > cable or two I think I can > get it all to work Thanks much for the tip. > > Clay > > > On Tue, May 6, 2014 at 11:00 AM, <users-requ...@open-mpi.org> wrote: > > > Send users mailing list submissions to > > us...@open-mpi.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > or, via email, send a message with subject or body 'help' to > > users-requ...@open-mpi.org > > > > You can reach the person managing the list at > > users-ow...@open-mpi.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of users digest..." > > > > > > Today's Topics: > > > > 1. Re: MPI_Barrier hangs on second attempt but only when > > multiple hosts used. (Daniels, Marcus G) > > 2. ROMIO bug reading darrays (Richard Shaw) > > 3. MPI File Open does not work (Imran Ali) > > 4. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > > 5. Re: MPI File Open does not work (Imran Ali) > > 6. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > > 7. Re: MPI File Open does not work (Imran Ali) > > 8. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > > 9. Re: users Digest, Vol 2879, Issue 1 (Jeff Squyres (jsquyres)) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Mon, 5 May 2014 19:28:07 +0000 > > From: "Daniels, Marcus G" <mdani...@lanl.gov> > > To: "'us...@open-mpi.org'" <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI_Barrier hangs on second attempt but only > > when multiple hosts used. > > Message-ID: > > < > > 532c594b7920a549a2a91cb4312cc57640dc5...@ecs-exg-p-mb01.win.lanl.gov> > > Content-Type: text/plain; charset="utf-8" > > > > > > > > From: Clay Kirkland [mailto:clay.kirkl...@versityinc.com] > > Sent: Friday, May 02, 2014 03:24 PM > > To: us...@open-mpi.org <us...@open-mpi.org> > > Subject: [OMPI users] MPI_Barrier hangs on second attempt but only when > > multiple hosts used. > > > > I have been using MPI for many many years so I have very well debugged mpi > > tests. I am > > having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions though > > with getting the > > MPI_Barrier calls to work. It works fine when I run all processes on one > > machine but when > > I run with two or more hosts the second call to MPI_Barrier always hangs. > > Not the first one, > > but always the second one. I looked at FAQ's and such but found nothing > > except for a comment > > that MPI_Barrier problems were often problems with fire walls. Also > > mentioned as a problem > > was not having the same version of mpi on both machines. I turned > > firewalls off and removed > > and reinstalled the same version on both hosts but I still see the same > > thing. I then installed > > lam mpi on two of my machines and that works fine. I can call the > > MPI_Barrier function when run on > > one of two machines by itself many times with no hangs. Only hangs if > > two or more hosts are involved. > > These runs are all being done on CentOS release 6.4. Here is test > > program I used. > > > > main (argc, argv) > > int argc; > > char **argv; > > { > > char message[20]; > > char hoster[256]; > > char nameis[256]; > > int fd, i, j, jnp, iret, myrank, np, ranker, recker; > > MPI_Comm comm; > > MPI_Status status; > > > > MPI_Init( &argc, &argv ); > > MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > > MPI_Comm_size( MPI_COMM_WORLD, &np); > > > > gethostname(hoster,256); > > > > printf(" In rank %d and host= %s Do Barrier call > > 1.\n",myrank,hoster); > > MPI_Barrier(MPI_COMM_WORLD); > > printf(" In rank %d and host= %s Do Barrier call > > 2.\n",myrank,hoster); > > MPI_Barrier(MPI_COMM_WORLD); > > printf(" In rank %d and host= %s Do Barrier call > > 3.\n",myrank,hoster); > > MPI_Barrier(MPI_COMM_WORLD); > > MPI_Finalize(); > > exit(0); > > } > > > > Here are three runs of test program. First with two processes on one > > host, then with > > two processes on another host, and finally with one process on each of two > > hosts. The > > first two runs are fine but the last run hangs on the second MPI_Barrier. > > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > > In rank 0 and host= centos Do Barrier call 1. > > In rank 1 and host= centos Do Barrier call 1. > > In rank 1 and host= centos Do Barrier call 2. > > In rank 1 and host= centos Do Barrier call 3. > > In rank 0 and host= centos Do Barrier call 2. > > In rank 0 and host= centos Do Barrier call 3. > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > > /root/.bashrc: line 14: unalias: ls: not found > > In rank 0 and host= RAID Do Barrier call 1. > > In rank 0 and host= RAID Do Barrier call 2. > > In rank 0 and host= RAID Do Barrier call 3. > > In rank 1 and host= RAID Do Barrier call 1. > > In rank 1 and host= RAID Do Barrier call 2. > > In rank 1 and host= RAID Do Barrier call 3. > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID a.out > > /root/.bashrc: line 14: unalias: ls: not found > > In rank 0 and host= centos Do Barrier call 1. > > In rank 0 and host= centos Do Barrier call 2. > > In rank 1 and host= RAID Do Barrier call 1. > > In rank 1 and host= RAID Do Barrier call 2. > > > > Since it is such a simple test and problem and such a widely used MPI > > function, it must obviously > > be an installation or configuration problem. A pstack for each of the > > hung MPI_Barrier processes > > on the two machines shows this: > > > > [root@centos ~]# pstack 31666 > > #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > > #1 0x00007f5de06125eb in epoll_dispatch () from /usr/local/lib/libmpi.so.1 > > #2 0x00007f5de061475a in opal_event_base_loop () from > > /usr/local/lib/libmpi.so.1 > > #3 0x00007f5de0639229 in opal_progress () from /usr/local/lib/libmpi.so.1 > > #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > > /usr/local/lib/libmpi.so.1 > > #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > #7 0x00007f5de05941c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > > #8 0x0000000000400a43 in main () > > > > [root@RAID openmpi-1.6.5]# pstack 22167 > > #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > > #1 0x00007f7ee46885eb in epoll_dispatch () from /usr/local/lib/libmpi.so.1 > > #2 0x00007f7ee468a75a in opal_event_base_loop () from > > /usr/local/lib/libmpi.so.1 > > #3 0x00007f7ee46af229 in opal_progress () from /usr/local/lib/libmpi.so.1 > > #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > > /usr/local/lib/libmpi.so.1 > > #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > #7 0x00007f7ee460a1c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > > #8 0x0000000000400a43 in main () > > > > Which looks exactly the same on each machine. Any thoughts or ideas > > would be greatly appreciated as > > I am stuck. > > > > Clay Kirkland > > > > > > > > > > > > > > > > > > -------------- next part -------------- > > HTML attachment scrubbed and removed > > > > ------------------------------ > > > > Message: 2 > > Date: Mon, 5 May 2014 22:20:59 -0400 > > From: Richard Shaw <jr...@cita.utoronto.ca> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: [OMPI users] ROMIO bug reading darrays > > Message-ID: > > < > > can+evmkc+9kacnpausscziufwdj3jfcsymb-8zdx1etdkab...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hello, > > > > I think I've come across a bug when using ROMIO to read in a 2D distributed > > array. I've attached a test case to this email. > > > > In the testcase I first initialise an array of 25 doubles (which will be a > > 5x5 grid), then I create a datatype representing a 5x5 matrix distributed > > in 3x3 blocks over a 2x2 process grid. As a reference I use MPI_Pack to > > pull out the block cyclic array elements local to each process (which I > > think is correct). Then I write the original array of 25 doubles to disk, > > and use MPI-IO to read it back in (performing the Open, Set_view, and > > Real_all), and compare to the reference. > > > > Running this with OMPI, the two match on all ranks. > > > > > mpirun -mca io ompio -np 4 ./darr_read.x > > === Rank 0 === (9 elements) > > Packed: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > > Read: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > > > > === Rank 1 === (6 elements) > > Packed: 15.0 16.0 17.0 20.0 21.0 22.0 > > Read: 15.0 16.0 17.0 20.0 21.0 22.0 > > > > === Rank 2 === (6 elements) > > Packed: 3.0 4.0 8.0 9.0 13.0 14.0 > > Read: 3.0 4.0 8.0 9.0 13.0 14.0 > > > > === Rank 3 === (4 elements) > > Packed: 18.0 19.0 23.0 24.0 > > Read: 18.0 19.0 23.0 24.0 > > > > > > > > However, using ROMIO the two differ on two of the ranks: > > > > > mpirun -mca io romio -np 4 ./darr_read.x > > === Rank 0 === (9 elements) > > Packed: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > > Read: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > > > > === Rank 1 === (6 elements) > > Packed: 15.0 16.0 17.0 20.0 21.0 22.0 > > Read: 0.0 1.0 2.0 0.0 1.0 2.0 > > > > === Rank 2 === (6 elements) > > Packed: 3.0 4.0 8.0 9.0 13.0 14.0 > > Read: 3.0 4.0 8.0 9.0 13.0 14.0 > > > > === Rank 3 === (4 elements) > > Packed: 18.0 19.0 23.0 24.0 > > Read: 0.0 1.0 0.0 1.0 > > > > > > > > My interpretation is that the behaviour with OMPIO is correct. > > Interestingly everything matches up using both ROMIO and OMPIO if I set the > > block shape to 2x2. > > > > This was run on OS X using 1.8.2a1r31632. I have also run this on Linux > > with OpenMPI 1.7.4, and OMPIO is still correct, but using ROMIO I just get > > segfaults. > > > > Thanks, > > Richard > > -------------- next part -------------- > > HTML attachment scrubbed and removed > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: darr_read.c > > Type: text/x-csrc > > Size: 2218 bytes > > Desc: not available > > URL: < > > http://www.open-mpi.org/MailArchives/users/attachments/20140505/5a5ab0ba/attachment.bin > > > > > > > ------------------------------ > > > > Message: 3 > > Date: Tue, 06 May 2014 13:24:35 +0200 > > From: Imran Ali <imra...@student.matnat.uio.no> > > To: <us...@open-mpi.org> > > Subject: [OMPI users] MPI File Open does not work > > Message-ID: <d57bdf499c00360b737205b085c50...@ulrik.uio.no> > > Content-Type: text/plain; charset="utf-8" > > > > > > > > I get the following error when I try to run the following python > > code > > import mpi4py.MPI as MPI > > comm = MPI.COMM_WORLD > > > > MPI.File.Open(comm,"some.file") > > > > $ mpirun -np 1 python > > test_mpi.py > > Traceback (most recent call last): > > File "test_mpi.py", line > > 3, in <module> > > MPI.File.Open(comm," h5ex_d_alloc.h5") > > File "File.pyx", > > line 67, in mpi4py.MPI.File.Open > > (src/mpi4py.MPI.c:89639) > > mpi4py.MPI.Exception: MPI_ERR_OTHER: known > > error not in > > list > > -------------------------------------------------------------------------- > > mpirun > > noticed that the job aborted, but has no info as to the process > > that > > caused that > > situation. > > -------------------------------------------------------------------------- > > > > > > My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > > dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > > (OS I am using, release 6.5) . It configured the build as following : > > > > > > ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > > --with-threads=posix --disable-mpi-profile > > > > I need emphasize that I do > > not have root acces on the system I am running my application. > > > > Imran > > > > > > > > -------------- next part -------------- > > HTML attachment scrubbed and removed > > > > ------------------------------ > > > > Message: 4 > > Date: Tue, 6 May 2014 12:56:04 +0000 > > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI File Open does not work > > Message-ID: <e7df28cb-d4fb-4087-928e-18e61d1d2...@cisco.com> > > Content-Type: text/plain; charset="us-ascii" > > > > The thread support in the 1.6 series is not very good. You might try: > > > > - Upgrading to 1.6.5 > > - Or better yet, upgrading to 1.8.1 > > > > > > On May 6, 2014, at 7:24 AM, Imran Ali <imra...@student.matnat.uio.no> > > wrote: > > > > > I get the following error when I try to run the following python code > > > > > > import mpi4py.MPI as MPI > > > comm = MPI.COMM_WORLD > > > MPI.File.Open(comm,"some.file") > > > > > > $ mpirun -np 1 python test_mpi.py > > > Traceback (most recent call last): > > > File "test_mpi.py", line 3, in <module> > > > MPI.File.Open(comm," h5ex_d_alloc.h5") > > > File "File.pyx", line 67, in mpi4py.MPI.File.Open > > (src/mpi4py.MPI.c:89639) > > > mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list > > > > > -------------------------------------------------------------------------- > > > mpirun noticed that the job aborted, but has no info as to the process > > > that caused that situation. > > > > > -------------------------------------------------------------------------- > > > > > > My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > > dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > > (OS I am using, release 6.5) . It configured the build as following : > > > > > > ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > > --with-threads=posix --disable-mpi-profile > > > > > > I need emphasize that I do not have root acces on the system I am > > running my application. > > > > > > Imran > > > > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > ------------------------------ > > > > Message: 5 > > Date: Tue, 6 May 2014 15:32:21 +0200 > > From: Imran Ali <imra...@student.matnat.uio.no> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI File Open does not work > > Message-ID: <fa6dffff-6c66-4a47-84fc-148fb51ce...@math.uio.no> > > Content-Type: text/plain; charset=us-ascii > > > > > > 6. mai 2014 kl. 14:56 skrev Jeff Squyres (jsquyres) <jsquy...@cisco.com>: > > > > > The thread support in the 1.6 series is not very good. You might try: > > > > > > - Upgrading to 1.6.5 > > > - Or better yet, upgrading to 1.8.1 > > > > > > > I will attempt that than. I read at > > > > http://www.open-mpi.org/faq/?category=building#install-overwrite > > > > that I should completely uninstall my previous version. Could you > > recommend to me how I can go about doing it (without root access). > > I am uncertain where I can use make uninstall. > > > > Imran > > > > > > > > On May 6, 2014, at 7:24 AM, Imran Ali <imra...@student.matnat.uio.no> > > wrote: > > > > > >> I get the following error when I try to run the following python code > > >> > > >> import mpi4py.MPI as MPI > > >> comm = MPI.COMM_WORLD > > >> MPI.File.Open(comm,"some.file") > > >> > > >> $ mpirun -np 1 python test_mpi.py > > >> Traceback (most recent call last): > > >> File "test_mpi.py", line 3, in <module> > > >> MPI.File.Open(comm," h5ex_d_alloc.h5") > > >> File "File.pyx", line 67, in mpi4py.MPI.File.Open > > (src/mpi4py.MPI.c:89639) > > >> mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list > > >> > > -------------------------------------------------------------------------- > > >> mpirun noticed that the job aborted, but has no info as to the process > > >> that caused that situation. > > >> > > -------------------------------------------------------------------------- > > >> > > >> My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > > dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > > (OS I am using, release 6.5) . It configured the build as following : > > >> > > >> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > > --with-threads=posix --disable-mpi-profile > > >> > > >> I need emphasize that I do not have root acces on the system I am > > running my application. > > >> > > >> Imran > > >> > > >> > > >> _______________________________________________ > > >> users mailing list > > >> us...@open-mpi.org > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > ------------------------------ > > > > Message: 6 > > Date: Tue, 6 May 2014 13:34:38 +0000 > > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI File Open does not work > > Message-ID: <2a933c0e-80f6-4ded-b44c-53b5f37ef...@cisco.com> > > Content-Type: text/plain; charset="us-ascii" > > > > On May 6, 2014, at 9:32 AM, Imran Ali <imra...@student.matnat.uio.no> > > wrote: > > > > > I will attempt that than. I read at > > > > > > http://www.open-mpi.org/faq/?category=building#install-overwrite > > > > > > that I should completely uninstall my previous version. > > > > Yes, that is best. OR: you can install into a whole separate tree and > > ignore the first installation. > > > > > Could you recommend to me how I can go about doing it (without root > > access). > > > I am uncertain where I can use make uninstall. > > > > If you don't have write access into the installation tree (i.e., it was > > installed via root and you don't have root access), then your best bet is > > simply to install into a new tree. E.g., if OMPI is installed into > > /opt/openmpi-1.6.2, try installing into /opt/openmpi-1.6.5, or even > > $HOME/installs/openmpi-1.6.5, or something like that. > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > ------------------------------ > > > > Message: 7 > > Date: Tue, 6 May 2014 15:40:34 +0200 > > From: Imran Ali <imra...@student.matnat.uio.no> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI File Open does not work > > Message-ID: <14f0596c-c5c5-4b1a-a4a8-8849d44ab...@math.uio.no> > > Content-Type: text/plain; charset=us-ascii > > > > > > 6. mai 2014 kl. 15:34 skrev Jeff Squyres (jsquyres) <jsquy...@cisco.com>: > > > > > On May 6, 2014, at 9:32 AM, Imran Ali <imra...@student.matnat.uio.no> > > wrote: > > > > > >> I will attempt that than. I read at > > >> > > >> http://www.open-mpi.org/faq/?category=building#install-overwrite > > >> > > >> that I should completely uninstall my previous version. > > > > > > Yes, that is best. OR: you can install into a whole separate tree and > > ignore the first installation. > > > > > >> Could you recommend to me how I can go about doing it (without root > > access). > > >> I am uncertain where I can use make uninstall. > > > > > > If you don't have write access into the installation tree (i.e., it was > > installed via root and you don't have root access), then your best bet is > > simply to install into a new tree. E.g., if OMPI is installed into > > /opt/openmpi-1.6.2, try installing into /opt/openmpi-1.6.5, or even > > $HOME/installs/openmpi-1.6.5, or something like that. > > > > My install was in my user directory (i.e $HOME). I managed to locate the > > source directory and successfully run make uninstall. > > > > Will let you know how things went after installation. > > > > Imran > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > ------------------------------ > > > > Message: 8 > > Date: Tue, 6 May 2014 14:42:52 +0000 > > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] MPI File Open does not work > > Message-ID: <710e3328-edaa-4a13-9f07-b45fe3191...@cisco.com> > > Content-Type: text/plain; charset="us-ascii" > > > > On May 6, 2014, at 9:40 AM, Imran Ali <imra...@student.matnat.uio.no> > > wrote: > > > > > My install was in my user directory (i.e $HOME). I managed to locate the > > source directory and successfully run make uninstall. > > > > > > FWIW, I usually install Open MPI into its own subdir. E.g., > > $HOME/installs/openmpi-x.y.z. Then if I don't want that install any more, > > I can just "rm -rf $HOME/installs/openmpi-x.y.z" -- no need to "make > > uninstall". Specifically: if there's nothing else installed in the same > > tree as Open MPI, you can just rm -rf its installation tree. > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > ------------------------------ > > > > Message: 9 > > Date: Tue, 6 May 2014 14:50:34 +0000 > > From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > > To: Open MPI Users <us...@open-mpi.org> > > Subject: Re: [OMPI users] users Digest, Vol 2879, Issue 1 > > Message-ID: <c60aa7e1-96a7-4ccd-9b5b-11a38fb87...@cisco.com> > > Content-Type: text/plain; charset="us-ascii" > > > > Are you using TCP as the MPI transport? > > > > If so, another thing to try is to limit the IP interfaces that MPI uses > > for its traffic to see if there's some kind of problem with specific > > networks. > > > > For example: > > > > mpirun --mca btl_tcp_if_include eth0 ... > > > > If that works, then try adding in any/all other IP interfaces that you > > have on your machines. > > > > A sorta-wild guess: you have some IP interfaces that aren't working, or at > > least, don't work in the way that OMPI wants them to work. So the first > > barrier works because it flows across eth0 (or some other first network > > that, as far as OMPI is concerned, works just fine). But then the next > > barrier round-robin advances to the next IP interface, and it doesn't work > > for some reason. > > > > We've seen virtual machine bridge interfaces cause problems, for example. > > E.g., if a machine has a Xen virtual machine interface (vibr0, IIRC?), > > then OMPI will try to use it to communicate with peer MPI processes because > > it has a "compatible" IP address, and OMPI think it should be > > connected/reachable to peers. If this is the case, you might want to > > disable such interfaces and/or use btl_tcp_if_include or btl_tcp_if_exclude > > to select the interfaces that you want to use. > > > > Pro tip: if you use btl_tcp_if_exclude, remember to exclude the loopback > > interface, too. OMPI defaults to a btl_tcp_if_include="" (blank) and > > btl_tcp_if_exclude="lo0". So if you override btl_tcp_if_exclude, you need > > to be sure to *also* include lo0 in the new value. For example: > > > > mpirun --mca btl_tcp_if_exclude lo0,virb0 ... > > > > Also, if possible, try upgrading to Open MPI 1.8.1. > > > > > > > > On May 4, 2014, at 2:15 PM, Clay Kirkland <clay.kirkl...@versityinc.com> > > wrote: > > > > > I am configuring with all defaults. Just doing a ./configure and then > > > make and make install. I have used open mpi on several kinds of > > > unix systems this way and have had no trouble before. I believe I > > > last had success on a redhat version of linux. > > > > > > > > > On Sat, May 3, 2014 at 11:00 AM, <users-requ...@open-mpi.org> wrote: > > > Send users mailing list submissions to > > > us...@open-mpi.org > > > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > or, via email, send a message with subject or body 'help' to > > > users-requ...@open-mpi.org > > > > > > You can reach the person managing the list at > > > users-ow...@open-mpi.org > > > > > > When replying, please edit your Subject line so it is more specific > > > than "Re: Contents of users digest..." > > > > > > > > > Today's Topics: > > > > > > 1. MPI_Barrier hangs on second attempt but only when multiple > > > hosts used. (Clay Kirkland) > > > 2. Re: MPI_Barrier hangs on second attempt but only when > > > multiple hosts used. (Ralph Castain) > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Message: 1 > > > Date: Fri, 2 May 2014 16:24:04 -0500 > > > From: Clay Kirkland <clay.kirkl...@versityinc.com> > > > To: us...@open-mpi.org > > > Subject: [OMPI users] MPI_Barrier hangs on second attempt but only > > > when multiple hosts used. > > > Message-ID: > > > <CAJDnjA8Wi=FEjz6Vz+Bc34b+nFE= > > tf4b7g0bqgmbekg7h-p...@mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > I have been using MPI for many many years so I have very well debugged > > mpi > > > tests. I am > > > having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions though > > > with getting the > > > MPI_Barrier calls to work. It works fine when I run all processes on > > one > > > machine but when > > > I run with two or more hosts the second call to MPI_Barrier always hangs. > > > Not the first one, > > > but always the second one. I looked at FAQ's and such but found nothing > > > except for a comment > > > that MPI_Barrier problems were often problems with fire walls. Also > > > mentioned as a problem > > > was not having the same version of mpi on both machines. I turned > > > firewalls off and removed > > > and reinstalled the same version on both hosts but I still see the same > > > thing. I then installed > > > lam mpi on two of my machines and that works fine. I can call the > > > MPI_Barrier function when run on > > > one of two machines by itself many times with no hangs. Only hangs if > > two > > > or more hosts are involved. > > > These runs are all being done on CentOS release 6.4. Here is test > > program > > > I used. > > > > > > main (argc, argv) > > > int argc; > > > char **argv; > > > { > > > char message[20]; > > > char hoster[256]; > > > char nameis[256]; > > > int fd, i, j, jnp, iret, myrank, np, ranker, recker; > > > MPI_Comm comm; > > > MPI_Status status; > > > > > > MPI_Init( &argc, &argv ); > > > MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > > > MPI_Comm_size( MPI_COMM_WORLD, &np); > > > > > > gethostname(hoster,256); > > > > > > printf(" In rank %d and host= %s Do Barrier call > > > 1.\n",myrank,hoster); > > > MPI_Barrier(MPI_COMM_WORLD); > > > printf(" In rank %d and host= %s Do Barrier call > > > 2.\n",myrank,hoster); > > > MPI_Barrier(MPI_COMM_WORLD); > > > printf(" In rank %d and host= %s Do Barrier call > > > 3.\n",myrank,hoster); > > > MPI_Barrier(MPI_COMM_WORLD); > > > MPI_Finalize(); > > > exit(0); > > > } > > > > > > Here are three runs of test program. First with two processes on one > > > host, then with > > > two processes on another host, and finally with one process on each of > > two > > > hosts. The > > > first two runs are fine but the last run hangs on the second MPI_Barrier. > > > > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > > > In rank 0 and host= centos Do Barrier call 1. > > > In rank 1 and host= centos Do Barrier call 1. > > > In rank 1 and host= centos Do Barrier call 2. > > > In rank 1 and host= centos Do Barrier call 3. > > > In rank 0 and host= centos Do Barrier call 2. > > > In rank 0 and host= centos Do Barrier call 3. > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > > > /root/.bashrc: line 14: unalias: ls: not found > > > In rank 0 and host= RAID Do Barrier call 1. > > > In rank 0 and host= RAID Do Barrier call 2. > > > In rank 0 and host= RAID Do Barrier call 3. > > > In rank 1 and host= RAID Do Barrier call 1. > > > In rank 1 and host= RAID Do Barrier call 2. > > > In rank 1 and host= RAID Do Barrier call 3. > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID a.out > > > /root/.bashrc: line 14: unalias: ls: not found > > > In rank 0 and host= centos Do Barrier call 1. > > > In rank 0 and host= centos Do Barrier call 2. > > > In rank 1 and host= RAID Do Barrier call 1. > > > In rank 1 and host= RAID Do Barrier call 2. > > > > > > Since it is such a simple test and problem and such a widely used MPI > > > function, it must obviously > > > be an installation or configuration problem. A pstack for each of the > > > hung MPI_Barrier processes > > > on the two machines shows this: > > > > > > [root@centos ~]# pstack 31666 > > > #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > > > #1 0x00007f5de06125eb in epoll_dispatch () from > > /usr/local/lib/libmpi.so.1 > > > #2 0x00007f5de061475a in opal_event_base_loop () from > > > /usr/local/lib/libmpi.so.1 > > > #3 0x00007f5de0639229 in opal_progress () from > > /usr/local/lib/libmpi.so.1 > > > #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > > > /usr/local/lib/libmpi.so.1 > > > #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () from > > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > #7 0x00007f5de05941c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > > > #8 0x0000000000400a43 in main () > > > > > > [root@RAID openmpi-1.6.5]# pstack 22167 > > > #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > > > #1 0x00007f7ee46885eb in epoll_dispatch () from > > /usr/local/lib/libmpi.so.1 > > > #2 0x00007f7ee468a75a in opal_event_base_loop () from > > > /usr/local/lib/libmpi.so.1 > > > #3 0x00007f7ee46af229 in opal_progress () from > > /usr/local/lib/libmpi.so.1 > > > #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > > > /usr/local/lib/libmpi.so.1 > > > #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () from > > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > #7 0x00007f7ee460a1c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > > > #8 0x0000000000400a43 in main () > > > > > > Which looks exactly the same on each machine. Any thoughts or ideas > > would > > > be greatly appreciated as > > > I am stuck. > > > > > > Clay Kirkland > > > -------------- next part -------------- > > > HTML attachment scrubbed and removed > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Sat, 3 May 2014 06:39:20 -0700 > > > From: Ralph Castain <r...@open-mpi.org> > > > To: Open MPI Users <us...@open-mpi.org> > > > Subject: Re: [OMPI users] MPI_Barrier hangs on second attempt but only > > > when multiple hosts used. > > > Message-ID: <3cf53d73-15d9-40bb-a2de-50ba3561a...@open-mpi.org> > > > Content-Type: text/plain; charset="us-ascii" > > > > > > Hmmm...just testing on my little cluster here on two nodes, it works > > just fine with 1.8.2: > > > > > > [rhc@bend001 v1.8]$ mpirun -n 2 --map-by node ./a.out > > > In rank 0 and host= bend001 Do Barrier call 1. > > > In rank 0 and host= bend001 Do Barrier call 2. > > > In rank 0 and host= bend001 Do Barrier call 3. > > > In rank 1 and host= bend002 Do Barrier call 1. > > > In rank 1 and host= bend002 Do Barrier call 2. > > > In rank 1 and host= bend002 Do Barrier call 3. > > > [rhc@bend001 v1.8]$ > > > > > > > > > How are you configuring OMPI? > > > > > > > > > On May 2, 2014, at 2:24 PM, Clay Kirkland <clay.kirkl...@versityinc.com> > > wrote: > > > > > > > I have been using MPI for many many years so I have very well debugged > > mpi tests. I am > > > > having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions > > though with getting the > > > > MPI_Barrier calls to work. It works fine when I run all processes on > > one machine but when > > > > I run with two or more hosts the second call to MPI_Barrier always > > hangs. Not the first one, > > > > but always the second one. I looked at FAQ's and such but found > > nothing except for a comment > > > > that MPI_Barrier problems were often problems with fire walls. Also > > mentioned as a problem > > > > was not having the same version of mpi on both machines. I turned > > firewalls off and removed > > > > and reinstalled the same version on both hosts but I still see the > > same thing. I then installed > > > > lam mpi on two of my machines and that works fine. I can call the > > MPI_Barrier function when run on > > > > one of two machines by itself many times with no hangs. Only hangs > > if two or more hosts are involved. > > > > These runs are all being done on CentOS release 6.4. Here is test > > program I used. > > > > > > > > main (argc, argv) > > > > int argc; > > > > char **argv; > > > > { > > > > char message[20]; > > > > char hoster[256]; > > > > char nameis[256]; > > > > int fd, i, j, jnp, iret, myrank, np, ranker, recker; > > > > MPI_Comm comm; > > > > MPI_Status status; > > > > > > > > MPI_Init( &argc, &argv ); > > > > MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > > > > MPI_Comm_size( MPI_COMM_WORLD, &np); > > > > > > > > gethostname(hoster,256); > > > > > > > > printf(" In rank %d and host= %s Do Barrier call > > 1.\n",myrank,hoster); > > > > MPI_Barrier(MPI_COMM_WORLD); > > > > printf(" In rank %d and host= %s Do Barrier call > > 2.\n",myrank,hoster); > > > > MPI_Barrier(MPI_COMM_WORLD); > > > > printf(" In rank %d and host= %s Do Barrier call > > 3.\n",myrank,hoster); > > > > MPI_Barrier(MPI_COMM_WORLD); > > > > MPI_Finalize(); > > > > exit(0); > > > > } > > > > > > > > Here are three runs of test program. First with two processes on > > one host, then with > > > > two processes on another host, and finally with one process on each of > > two hosts. The > > > > first two runs are fine but the last run hangs on the second > > MPI_Barrier. > > > > > > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > > > > In rank 0 and host= centos Do Barrier call 1. > > > > In rank 1 and host= centos Do Barrier call 1. > > > > In rank 1 and host= centos Do Barrier call 2. > > > > In rank 1 and host= centos Do Barrier call 3. > > > > In rank 0 and host= centos Do Barrier call 2. > > > > In rank 0 and host= centos Do Barrier call 3. > > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > > > > /root/.bashrc: line 14: unalias: ls: not found > > > > In rank 0 and host= RAID Do Barrier call 1. > > > > In rank 0 and host= RAID Do Barrier call 2. > > > > In rank 0 and host= RAID Do Barrier call 3. > > > > In rank 1 and host= RAID Do Barrier call 1. > > > > In rank 1 and host= RAID Do Barrier call 2. > > > > In rank 1 and host= RAID Do Barrier call 3. > > > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID > > a.out > > > > /root/.bashrc: line 14: unalias: ls: not found > > > > In rank 0 and host= centos Do Barrier call 1. > > > > In rank 0 and host= centos Do Barrier call 2. > > > > In rank 1 and host= RAID Do Barrier call 1. > > > > In rank 1 and host= RAID Do Barrier call 2. > > > > > > > > Since it is such a simple test and problem and such a widely used > > MPI function, it must obviously > > > > be an installation or configuration problem. A pstack for each of > > the hung MPI_Barrier processes > > > > on the two machines shows this: > > > > > > > > [root@centos ~]# pstack 31666 > > > > #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from > > /lib64/libc.so.6 > > > > #1 0x00007f5de06125eb in epoll_dispatch () from > > /usr/local/lib/libmpi.so.1 > > > > #2 0x00007f5de061475a in opal_event_base_loop () from > > /usr/local/lib/libmpi.so.1 > > > > #3 0x00007f5de0639229 in opal_progress () from > > /usr/local/lib/libmpi.so.1 > > > > #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > > /usr/local/lib/libmpi.so.1 > > > > #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > > #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () > > from /usr/local/lib/openmpi/mca_coll_tuned.so > > > > #7 0x00007f5de05941c2 in PMPI_Barrier () from > > /usr/local/lib/libmpi.so.1 > > > > #8 0x0000000000400a43 in main () > > > > > > > > [root@RAID openmpi-1.6.5]# pstack 22167 > > > > #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from > > /lib64/libc.so.6 > > > > #1 0x00007f7ee46885eb in epoll_dispatch () from > > /usr/local/lib/libmpi.so.1 > > > > #2 0x00007f7ee468a75a in opal_event_base_loop () from > > /usr/local/lib/libmpi.so.1 > > > > #3 0x00007f7ee46af229 in opal_progress () from > > /usr/local/lib/libmpi.so.1 > > > > #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > > /usr/local/lib/libmpi.so.1 > > > > #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > > /usr/local/lib/openmpi/mca_coll_tuned.so > > > > #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () > > from /usr/local/lib/openmpi/mca_coll_tuned.so > > > > #7 0x00007f7ee460a1c2 in PMPI_Barrier () from > > /usr/local/lib/libmpi.so.1 > > > > #8 0x0000000000400a43 in main () > > > > > > > > Which looks exactly the same on each machine. Any thoughts or ideas > > would be greatly appreciated as > > > > I am stuck. > > > > > > > > Clay Kirkland > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -------------- next part -------------- > > > HTML attachment scrubbed and removed > > > > > > ------------------------------ > > > > > > Subject: Digest Footer > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > ------------------------------ > > > > > > End of users Digest, Vol 2879, Issue 1 > > > ************************************** > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ------------------------------ > > > > End of users Digest, Vol 2881, Issue 1 > > ************************************** > > > -------------- next part -------------- > HTML attachment scrubbed and removed > > ------------------------------ > > Message: 2 > Date: Tue, 6 May 2014 18:50:50 -0500 > From: Clay Kirkland <clay.kirkl...@versityinc.com> > To: us...@open-mpi.org > Subject: Re: [OMPI users] users Digest, Vol 2881, Issue 1 > Message-ID: > <cajdnja-u4btpto+87czsho81t+-a1jzottc7mwdfiar7+vz...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Well it turns out I can't seem to get all three of my machines on the > same page. > Two of them are using eth0 and one is using eth1. Centos seems unable to > bring > up multiple network interfaces for some reason and when I use the mca param > to > use eth0 it works on two machines but not the other. Is there some way to > use > only eth1 on one host and only eth0 on the other two? Maybe environment > variables > but I can't seem to get that to work either. > > Clay > > > On Tue, May 6, 2014 at 6:28 PM, Clay Kirkland > <clay.kirkl...@versityinc.com>wrote: > > > That last trick seems to work. I can get it to work once in a while with > > those tcp options but it is > > tricky as I have three machines and two of them use eth0 as primary > > network interface and one > > uses eth1. But by fiddling with network options and perhaps moving a > > cable or two I think I can > > get it all to work Thanks much for the tip. > > > > Clay > > > > > > On Tue, May 6, 2014 at 11:00 AM, <users-requ...@open-mpi.org> wrote: > > > >> Send users mailing list submissions to > >> us...@open-mpi.org > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> or, via email, send a message with subject or body 'help' to > >> users-requ...@open-mpi.org > >> > >> You can reach the person managing the list at > >> users-ow...@open-mpi.org > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of users digest..." > >> > >> > >> Today's Topics: > >> > >> 1. Re: MPI_Barrier hangs on second attempt but only when > >> multiple hosts used. (Daniels, Marcus G) > >> 2. ROMIO bug reading darrays (Richard Shaw) > >> 3. MPI File Open does not work (Imran Ali) > >> 4. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > >> 5. Re: MPI File Open does not work (Imran Ali) > >> 6. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > >> 7. Re: MPI File Open does not work (Imran Ali) > >> 8. Re: MPI File Open does not work (Jeff Squyres (jsquyres)) > >> 9. Re: users Digest, Vol 2879, Issue 1 (Jeff Squyres (jsquyres)) > >> > >> > >> ---------------------------------------------------------------------- > >> > >> Message: 1 > >> Date: Mon, 5 May 2014 19:28:07 +0000 > >> From: "Daniels, Marcus G" <mdani...@lanl.gov> > >> To: "'us...@open-mpi.org'" <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI_Barrier hangs on second attempt but only > >> when multiple hosts used. > >> Message-ID: > >> < > >> 532c594b7920a549a2a91cb4312cc57640dc5...@ecs-exg-p-mb01.win.lanl.gov> > >> Content-Type: text/plain; charset="utf-8" > >> > >> > >> > >> From: Clay Kirkland [mailto:clay.kirkl...@versityinc.com] > >> Sent: Friday, May 02, 2014 03:24 PM > >> To: us...@open-mpi.org <us...@open-mpi.org> > >> Subject: [OMPI users] MPI_Barrier hangs on second attempt but only when > >> multiple hosts used. > >> > >> I have been using MPI for many many years so I have very well debugged > >> mpi tests. I am > >> having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions though > >> with getting the > >> MPI_Barrier calls to work. It works fine when I run all processes on > >> one machine but when > >> I run with two or more hosts the second call to MPI_Barrier always hangs. > >> Not the first one, > >> but always the second one. I looked at FAQ's and such but found nothing > >> except for a comment > >> that MPI_Barrier problems were often problems with fire walls. Also > >> mentioned as a problem > >> was not having the same version of mpi on both machines. I turned > >> firewalls off and removed > >> and reinstalled the same version on both hosts but I still see the same > >> thing. I then installed > >> lam mpi on two of my machines and that works fine. I can call the > >> MPI_Barrier function when run on > >> one of two machines by itself many times with no hangs. Only hangs if > >> two or more hosts are involved. > >> These runs are all being done on CentOS release 6.4. Here is test > >> program I used. > >> > >> main (argc, argv) > >> int argc; > >> char **argv; > >> { > >> char message[20]; > >> char hoster[256]; > >> char nameis[256]; > >> int fd, i, j, jnp, iret, myrank, np, ranker, recker; > >> MPI_Comm comm; > >> MPI_Status status; > >> > >> MPI_Init( &argc, &argv ); > >> MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > >> MPI_Comm_size( MPI_COMM_WORLD, &np); > >> > >> gethostname(hoster,256); > >> > >> printf(" In rank %d and host= %s Do Barrier call > >> 1.\n",myrank,hoster); > >> MPI_Barrier(MPI_COMM_WORLD); > >> printf(" In rank %d and host= %s Do Barrier call > >> 2.\n",myrank,hoster); > >> MPI_Barrier(MPI_COMM_WORLD); > >> printf(" In rank %d and host= %s Do Barrier call > >> 3.\n",myrank,hoster); > >> MPI_Barrier(MPI_COMM_WORLD); > >> MPI_Finalize(); > >> exit(0); > >> } > >> > >> Here are three runs of test program. First with two processes on one > >> host, then with > >> two processes on another host, and finally with one process on each of > >> two hosts. The > >> first two runs are fine but the last run hangs on the second MPI_Barrier. > >> > >> [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > >> In rank 0 and host= centos Do Barrier call 1. > >> In rank 1 and host= centos Do Barrier call 1. > >> In rank 1 and host= centos Do Barrier call 2. > >> In rank 1 and host= centos Do Barrier call 3. > >> In rank 0 and host= centos Do Barrier call 2. > >> In rank 0 and host= centos Do Barrier call 3. > >> [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > >> /root/.bashrc: line 14: unalias: ls: not found > >> In rank 0 and host= RAID Do Barrier call 1. > >> In rank 0 and host= RAID Do Barrier call 2. > >> In rank 0 and host= RAID Do Barrier call 3. > >> In rank 1 and host= RAID Do Barrier call 1. > >> In rank 1 and host= RAID Do Barrier call 2. > >> In rank 1 and host= RAID Do Barrier call 3. > >> [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID a.out > >> /root/.bashrc: line 14: unalias: ls: not found > >> In rank 0 and host= centos Do Barrier call 1. > >> In rank 0 and host= centos Do Barrier call 2. > >> In rank 1 and host= RAID Do Barrier call 1. > >> In rank 1 and host= RAID Do Barrier call 2. > >> > >> Since it is such a simple test and problem and such a widely used MPI > >> function, it must obviously > >> be an installation or configuration problem. A pstack for each of the > >> hung MPI_Barrier processes > >> on the two machines shows this: > >> > >> [root@centos ~]# pstack 31666 > >> #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > >> #1 0x00007f5de06125eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> #2 0x00007f5de061475a in opal_event_base_loop () from > >> /usr/local/lib/libmpi.so.1 > >> #3 0x00007f5de0639229 in opal_progress () from /usr/local/lib/libmpi.so.1 > >> #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > >> /usr/local/lib/libmpi.so.1 > >> #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> #7 0x00007f5de05941c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > >> #8 0x0000000000400a43 in main () > >> > >> [root@RAID openmpi-1.6.5]# pstack 22167 > >> #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > >> #1 0x00007f7ee46885eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> #2 0x00007f7ee468a75a in opal_event_base_loop () from > >> /usr/local/lib/libmpi.so.1 > >> #3 0x00007f7ee46af229 in opal_progress () from /usr/local/lib/libmpi.so.1 > >> #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > >> /usr/local/lib/libmpi.so.1 > >> #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> #7 0x00007f7ee460a1c2 in PMPI_Barrier () from /usr/local/lib/libmpi.so.1 > >> #8 0x0000000000400a43 in main () > >> > >> Which looks exactly the same on each machine. Any thoughts or ideas > >> would be greatly appreciated as > >> I am stuck. > >> > >> Clay Kirkland > >> > >> > >> > >> > >> > >> > >> > >> > >> -------------- next part -------------- > >> HTML attachment scrubbed and removed > >> > >> ------------------------------ > >> > >> Message: 2 > >> Date: Mon, 5 May 2014 22:20:59 -0400 > >> From: Richard Shaw <jr...@cita.utoronto.ca> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: [OMPI users] ROMIO bug reading darrays > >> Message-ID: > >> < > >> can+evmkc+9kacnpausscziufwdj3jfcsymb-8zdx1etdkab...@mail.gmail.com> > >> Content-Type: text/plain; charset="utf-8" > >> > >> Hello, > >> > >> I think I've come across a bug when using ROMIO to read in a 2D > >> distributed > >> array. I've attached a test case to this email. > >> > >> In the testcase I first initialise an array of 25 doubles (which will be a > >> 5x5 grid), then I create a datatype representing a 5x5 matrix distributed > >> in 3x3 blocks over a 2x2 process grid. As a reference I use MPI_Pack to > >> pull out the block cyclic array elements local to each process (which I > >> think is correct). Then I write the original array of 25 doubles to disk, > >> and use MPI-IO to read it back in (performing the Open, Set_view, and > >> Real_all), and compare to the reference. > >> > >> Running this with OMPI, the two match on all ranks. > >> > >> > mpirun -mca io ompio -np 4 ./darr_read.x > >> === Rank 0 === (9 elements) > >> Packed: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > >> Read: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > >> > >> === Rank 1 === (6 elements) > >> Packed: 15.0 16.0 17.0 20.0 21.0 22.0 > >> Read: 15.0 16.0 17.0 20.0 21.0 22.0 > >> > >> === Rank 2 === (6 elements) > >> Packed: 3.0 4.0 8.0 9.0 13.0 14.0 > >> Read: 3.0 4.0 8.0 9.0 13.0 14.0 > >> > >> === Rank 3 === (4 elements) > >> Packed: 18.0 19.0 23.0 24.0 > >> Read: 18.0 19.0 23.0 24.0 > >> > >> > >> > >> However, using ROMIO the two differ on two of the ranks: > >> > >> > mpirun -mca io romio -np 4 ./darr_read.x > >> === Rank 0 === (9 elements) > >> Packed: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > >> Read: 0.0 1.0 2.0 5.0 6.0 7.0 10.0 11.0 12.0 > >> > >> === Rank 1 === (6 elements) > >> Packed: 15.0 16.0 17.0 20.0 21.0 22.0 > >> Read: 0.0 1.0 2.0 0.0 1.0 2.0 > >> > >> === Rank 2 === (6 elements) > >> Packed: 3.0 4.0 8.0 9.0 13.0 14.0 > >> Read: 3.0 4.0 8.0 9.0 13.0 14.0 > >> > >> === Rank 3 === (4 elements) > >> Packed: 18.0 19.0 23.0 24.0 > >> Read: 0.0 1.0 0.0 1.0 > >> > >> > >> > >> My interpretation is that the behaviour with OMPIO is correct. > >> Interestingly everything matches up using both ROMIO and OMPIO if I set > >> the > >> block shape to 2x2. > >> > >> This was run on OS X using 1.8.2a1r31632. I have also run this on Linux > >> with OpenMPI 1.7.4, and OMPIO is still correct, but using ROMIO I just get > >> segfaults. > >> > >> Thanks, > >> Richard > >> -------------- next part -------------- > >> HTML attachment scrubbed and removed > >> -------------- next part -------------- > >> A non-text attachment was scrubbed... > >> Name: darr_read.c > >> Type: text/x-csrc > >> Size: 2218 bytes > >> Desc: not available > >> URL: < > >> http://www.open-mpi.org/MailArchives/users/attachments/20140505/5a5ab0ba/attachment.bin > >> > > >> > >> ------------------------------ > >> > >> Message: 3 > >> Date: Tue, 06 May 2014 13:24:35 +0200 > >> From: Imran Ali <imra...@student.matnat.uio.no> > >> To: <us...@open-mpi.org> > >> Subject: [OMPI users] MPI File Open does not work > >> Message-ID: <d57bdf499c00360b737205b085c50...@ulrik.uio.no> > >> Content-Type: text/plain; charset="utf-8" > >> > >> > >> > >> I get the following error when I try to run the following python > >> code > >> import mpi4py.MPI as MPI > >> comm = MPI.COMM_WORLD > >> > >> MPI.File.Open(comm,"some.file") > >> > >> $ mpirun -np 1 python > >> test_mpi.py > >> Traceback (most recent call last): > >> File "test_mpi.py", line > >> 3, in <module> > >> MPI.File.Open(comm," h5ex_d_alloc.h5") > >> File "File.pyx", > >> line 67, in mpi4py.MPI.File.Open > >> (src/mpi4py.MPI.c:89639) > >> mpi4py.MPI.Exception: MPI_ERR_OTHER: known > >> error not in > >> list > >> -------------------------------------------------------------------------- > >> mpirun > >> noticed that the job aborted, but has no info as to the process > >> that > >> caused that > >> situation. > >> -------------------------------------------------------------------------- > >> > >> > >> My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > >> dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > >> (OS I am using, release 6.5) . It configured the build as following : > >> > >> > >> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > >> --with-threads=posix --disable-mpi-profile > >> > >> I need emphasize that I do > >> not have root acces on the system I am running my application. > >> > >> Imran > >> > >> > >> > >> -------------- next part -------------- > >> HTML attachment scrubbed and removed > >> > >> ------------------------------ > >> > >> Message: 4 > >> Date: Tue, 6 May 2014 12:56:04 +0000 > >> From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI File Open does not work > >> Message-ID: <e7df28cb-d4fb-4087-928e-18e61d1d2...@cisco.com> > >> Content-Type: text/plain; charset="us-ascii" > >> > >> The thread support in the 1.6 series is not very good. You might try: > >> > >> - Upgrading to 1.6.5 > >> - Or better yet, upgrading to 1.8.1 > >> > >> > >> On May 6, 2014, at 7:24 AM, Imran Ali <imra...@student.matnat.uio.no> > >> wrote: > >> > >> > I get the following error when I try to run the following python code > >> > > >> > import mpi4py.MPI as MPI > >> > comm = MPI.COMM_WORLD > >> > MPI.File.Open(comm,"some.file") > >> > > >> > $ mpirun -np 1 python test_mpi.py > >> > Traceback (most recent call last): > >> > File "test_mpi.py", line 3, in <module> > >> > MPI.File.Open(comm," h5ex_d_alloc.h5") > >> > File "File.pyx", line 67, in mpi4py.MPI.File.Open > >> (src/mpi4py.MPI.c:89639) > >> > mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list > >> > > >> -------------------------------------------------------------------------- > >> > mpirun noticed that the job aborted, but has no info as to the process > >> > that caused that situation. > >> > > >> -------------------------------------------------------------------------- > >> > > >> > My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > >> dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > >> (OS I am using, release 6.5) . It configured the build as following : > >> > > >> > ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > >> --with-threads=posix --disable-mpi-profile > >> > > >> > I need emphasize that I do not have root acces on the system I am > >> running my application. > >> > > >> > Imran > >> > > >> > > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> > >> > >> ------------------------------ > >> > >> Message: 5 > >> Date: Tue, 6 May 2014 15:32:21 +0200 > >> From: Imran Ali <imra...@student.matnat.uio.no> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI File Open does not work > >> Message-ID: <fa6dffff-6c66-4a47-84fc-148fb51ce...@math.uio.no> > >> Content-Type: text/plain; charset=us-ascii > >> > >> > >> 6. mai 2014 kl. 14:56 skrev Jeff Squyres (jsquyres) <jsquy...@cisco.com>: > >> > >> > The thread support in the 1.6 series is not very good. You might try: > >> > > >> > - Upgrading to 1.6.5 > >> > - Or better yet, upgrading to 1.8.1 > >> > > >> > >> I will attempt that than. I read at > >> > >> http://www.open-mpi.org/faq/?category=building#install-overwrite > >> > >> that I should completely uninstall my previous version. Could you > >> recommend to me how I can go about doing it (without root access). > >> I am uncertain where I can use make uninstall. > >> > >> Imran > >> > >> > > >> > On May 6, 2014, at 7:24 AM, Imran Ali <imra...@student.matnat.uio.no> > >> wrote: > >> > > >> >> I get the following error when I try to run the following python code > >> >> > >> >> import mpi4py.MPI as MPI > >> >> comm = MPI.COMM_WORLD > >> >> MPI.File.Open(comm,"some.file") > >> >> > >> >> $ mpirun -np 1 python test_mpi.py > >> >> Traceback (most recent call last): > >> >> File "test_mpi.py", line 3, in <module> > >> >> MPI.File.Open(comm," h5ex_d_alloc.h5") > >> >> File "File.pyx", line 67, in mpi4py.MPI.File.Open > >> (src/mpi4py.MPI.c:89639) > >> >> mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list > >> >> > >> -------------------------------------------------------------------------- > >> >> mpirun noticed that the job aborted, but has no info as to the process > >> >> that caused that situation. > >> >> > >> -------------------------------------------------------------------------- > >> >> > >> >> My mpirun version is (Open MPI) 1.6.2. I installed openmpi using the > >> dorsal script (https://github.com/FEniCS/dorsal) for Redhat Enterprise 6 > >> (OS I am using, release 6.5) . It configured the build as following : > >> >> > >> >> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads > >> --with-threads=posix --disable-mpi-profile > >> >> > >> >> I need emphasize that I do not have root acces on the system I am > >> running my application. > >> >> > >> >> Imran > >> >> > >> >> > >> >> _______________________________________________ > >> >> users mailing list > >> >> us...@open-mpi.org > >> >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > > >> > > >> > -- > >> > Jeff Squyres > >> > jsquy...@cisco.com > >> > For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> > >> ------------------------------ > >> > >> Message: 6 > >> Date: Tue, 6 May 2014 13:34:38 +0000 > >> From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI File Open does not work > >> Message-ID: <2a933c0e-80f6-4ded-b44c-53b5f37ef...@cisco.com> > >> Content-Type: text/plain; charset="us-ascii" > >> > >> On May 6, 2014, at 9:32 AM, Imran Ali <imra...@student.matnat.uio.no> > >> wrote: > >> > >> > I will attempt that than. I read at > >> > > >> > http://www.open-mpi.org/faq/?category=building#install-overwrite > >> > > >> > that I should completely uninstall my previous version. > >> > >> Yes, that is best. OR: you can install into a whole separate tree and > >> ignore the first installation. > >> > >> > Could you recommend to me how I can go about doing it (without root > >> access). > >> > I am uncertain where I can use make uninstall. > >> > >> If you don't have write access into the installation tree (i.e., it was > >> installed via root and you don't have root access), then your best bet is > >> simply to install into a new tree. E.g., if OMPI is installed into > >> /opt/openmpi-1.6.2, try installing into /opt/openmpi-1.6.5, or even > >> $HOME/installs/openmpi-1.6.5, or something like that. > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> > >> > >> ------------------------------ > >> > >> Message: 7 > >> Date: Tue, 6 May 2014 15:40:34 +0200 > >> From: Imran Ali <imra...@student.matnat.uio.no> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI File Open does not work > >> Message-ID: <14f0596c-c5c5-4b1a-a4a8-8849d44ab...@math.uio.no> > >> Content-Type: text/plain; charset=us-ascii > >> > >> > >> 6. mai 2014 kl. 15:34 skrev Jeff Squyres (jsquyres) <jsquy...@cisco.com>: > >> > >> > On May 6, 2014, at 9:32 AM, Imran Ali <imra...@student.matnat.uio.no> > >> wrote: > >> > > >> >> I will attempt that than. I read at > >> >> > >> >> http://www.open-mpi.org/faq/?category=building#install-overwrite > >> >> > >> >> that I should completely uninstall my previous version. > >> > > >> > Yes, that is best. OR: you can install into a whole separate tree and > >> ignore the first installation. > >> > > >> >> Could you recommend to me how I can go about doing it (without root > >> access). > >> >> I am uncertain where I can use make uninstall. > >> > > >> > If you don't have write access into the installation tree (i.e., it was > >> installed via root and you don't have root access), then your best bet is > >> simply to install into a new tree. E.g., if OMPI is installed into > >> /opt/openmpi-1.6.2, try installing into /opt/openmpi-1.6.5, or even > >> $HOME/installs/openmpi-1.6.5, or something like that. > >> > >> My install was in my user directory (i.e $HOME). I managed to locate the > >> source directory and successfully run make uninstall. > >> > >> Will let you know how things went after installation. > >> > >> Imran > >> > >> > > >> > -- > >> > Jeff Squyres > >> > jsquy...@cisco.com > >> > For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> > >> ------------------------------ > >> > >> Message: 8 > >> Date: Tue, 6 May 2014 14:42:52 +0000 > >> From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] MPI File Open does not work > >> Message-ID: <710e3328-edaa-4a13-9f07-b45fe3191...@cisco.com> > >> Content-Type: text/plain; charset="us-ascii" > >> > >> On May 6, 2014, at 9:40 AM, Imran Ali <imra...@student.matnat.uio.no> > >> wrote: > >> > >> > My install was in my user directory (i.e $HOME). I managed to locate > >> the source directory and successfully run make uninstall. > >> > >> > >> FWIW, I usually install Open MPI into its own subdir. E.g., > >> $HOME/installs/openmpi-x.y.z. Then if I don't want that install any more, > >> I can just "rm -rf $HOME/installs/openmpi-x.y.z" -- no need to "make > >> uninstall". Specifically: if there's nothing else installed in the same > >> tree as Open MPI, you can just rm -rf its installation tree. > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> > >> > >> ------------------------------ > >> > >> Message: 9 > >> Date: Tue, 6 May 2014 14:50:34 +0000 > >> From: "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> > >> To: Open MPI Users <us...@open-mpi.org> > >> Subject: Re: [OMPI users] users Digest, Vol 2879, Issue 1 > >> Message-ID: <c60aa7e1-96a7-4ccd-9b5b-11a38fb87...@cisco.com> > >> Content-Type: text/plain; charset="us-ascii" > >> > >> Are you using TCP as the MPI transport? > >> > >> If so, another thing to try is to limit the IP interfaces that MPI uses > >> for its traffic to see if there's some kind of problem with specific > >> networks. > >> > >> For example: > >> > >> mpirun --mca btl_tcp_if_include eth0 ... > >> > >> If that works, then try adding in any/all other IP interfaces that you > >> have on your machines. > >> > >> A sorta-wild guess: you have some IP interfaces that aren't working, or > >> at least, don't work in the way that OMPI wants them to work. So the first > >> barrier works because it flows across eth0 (or some other first network > >> that, as far as OMPI is concerned, works just fine). But then the next > >> barrier round-robin advances to the next IP interface, and it doesn't work > >> for some reason. > >> > >> We've seen virtual machine bridge interfaces cause problems, for example. > >> E.g., if a machine has a Xen virtual machine interface (vibr0, IIRC?), > >> then OMPI will try to use it to communicate with peer MPI processes because > >> it has a "compatible" IP address, and OMPI think it should be > >> connected/reachable to peers. If this is the case, you might want to > >> disable such interfaces and/or use btl_tcp_if_include or btl_tcp_if_exclude > >> to select the interfaces that you want to use. > >> > >> Pro tip: if you use btl_tcp_if_exclude, remember to exclude the loopback > >> interface, too. OMPI defaults to a btl_tcp_if_include="" (blank) and > >> btl_tcp_if_exclude="lo0". So if you override btl_tcp_if_exclude, you need > >> to be sure to *also* include lo0 in the new value. For example: > >> > >> mpirun --mca btl_tcp_if_exclude lo0,virb0 ... > >> > >> Also, if possible, try upgrading to Open MPI 1.8.1. > >> > >> > >> > >> On May 4, 2014, at 2:15 PM, Clay Kirkland <clay.kirkl...@versityinc.com> > >> wrote: > >> > >> > I am configuring with all defaults. Just doing a ./configure and then > >> > make and make install. I have used open mpi on several kinds of > >> > unix systems this way and have had no trouble before. I believe I > >> > last had success on a redhat version of linux. > >> > > >> > > >> > On Sat, May 3, 2014 at 11:00 AM, <users-requ...@open-mpi.org> wrote: > >> > Send users mailing list submissions to > >> > us...@open-mpi.org > >> > > >> > To subscribe or unsubscribe via the World Wide Web, visit > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > or, via email, send a message with subject or body 'help' to > >> > users-requ...@open-mpi.org > >> > > >> > You can reach the person managing the list at > >> > users-ow...@open-mpi.org > >> > > >> > When replying, please edit your Subject line so it is more specific > >> > than "Re: Contents of users digest..." > >> > > >> > > >> > Today's Topics: > >> > > >> > 1. MPI_Barrier hangs on second attempt but only when multiple > >> > hosts used. (Clay Kirkland) > >> > 2. Re: MPI_Barrier hangs on second attempt but only when > >> > multiple hosts used. (Ralph Castain) > >> > > >> > > >> > ---------------------------------------------------------------------- > >> > > >> > Message: 1 > >> > Date: Fri, 2 May 2014 16:24:04 -0500 > >> > From: Clay Kirkland <clay.kirkl...@versityinc.com> > >> > To: us...@open-mpi.org > >> > Subject: [OMPI users] MPI_Barrier hangs on second attempt but only > >> > when multiple hosts used. > >> > Message-ID: > >> > <CAJDnjA8Wi=FEjz6Vz+Bc34b+nFE= > >> tf4b7g0bqgmbekg7h-p...@mail.gmail.com> > >> > Content-Type: text/plain; charset="utf-8" > >> > > >> > I have been using MPI for many many years so I have very well debugged > >> mpi > >> > tests. I am > >> > having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions > >> though > >> > with getting the > >> > MPI_Barrier calls to work. It works fine when I run all processes on > >> one > >> > machine but when > >> > I run with two or more hosts the second call to MPI_Barrier always > >> hangs. > >> > Not the first one, > >> > but always the second one. I looked at FAQ's and such but found > >> nothing > >> > except for a comment > >> > that MPI_Barrier problems were often problems with fire walls. Also > >> > mentioned as a problem > >> > was not having the same version of mpi on both machines. I turned > >> > firewalls off and removed > >> > and reinstalled the same version on both hosts but I still see the same > >> > thing. I then installed > >> > lam mpi on two of my machines and that works fine. I can call the > >> > MPI_Barrier function when run on > >> > one of two machines by itself many times with no hangs. Only hangs if > >> two > >> > or more hosts are involved. > >> > These runs are all being done on CentOS release 6.4. Here is test > >> program > >> > I used. > >> > > >> > main (argc, argv) > >> > int argc; > >> > char **argv; > >> > { > >> > char message[20]; > >> > char hoster[256]; > >> > char nameis[256]; > >> > int fd, i, j, jnp, iret, myrank, np, ranker, recker; > >> > MPI_Comm comm; > >> > MPI_Status status; > >> > > >> > MPI_Init( &argc, &argv ); > >> > MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > >> > MPI_Comm_size( MPI_COMM_WORLD, &np); > >> > > >> > gethostname(hoster,256); > >> > > >> > printf(" In rank %d and host= %s Do Barrier call > >> > 1.\n",myrank,hoster); > >> > MPI_Barrier(MPI_COMM_WORLD); > >> > printf(" In rank %d and host= %s Do Barrier call > >> > 2.\n",myrank,hoster); > >> > MPI_Barrier(MPI_COMM_WORLD); > >> > printf(" In rank %d and host= %s Do Barrier call > >> > 3.\n",myrank,hoster); > >> > MPI_Barrier(MPI_COMM_WORLD); > >> > MPI_Finalize(); > >> > exit(0); > >> > } > >> > > >> > Here are three runs of test program. First with two processes on one > >> > host, then with > >> > two processes on another host, and finally with one process on each of > >> two > >> > hosts. The > >> > first two runs are fine but the last run hangs on the second > >> MPI_Barrier. > >> > > >> > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > >> > In rank 0 and host= centos Do Barrier call 1. > >> > In rank 1 and host= centos Do Barrier call 1. > >> > In rank 1 and host= centos Do Barrier call 2. > >> > In rank 1 and host= centos Do Barrier call 3. > >> > In rank 0 and host= centos Do Barrier call 2. > >> > In rank 0 and host= centos Do Barrier call 3. > >> > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > >> > /root/.bashrc: line 14: unalias: ls: not found > >> > In rank 0 and host= RAID Do Barrier call 1. > >> > In rank 0 and host= RAID Do Barrier call 2. > >> > In rank 0 and host= RAID Do Barrier call 3. > >> > In rank 1 and host= RAID Do Barrier call 1. > >> > In rank 1 and host= RAID Do Barrier call 2. > >> > In rank 1 and host= RAID Do Barrier call 3. > >> > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID a.out > >> > /root/.bashrc: line 14: unalias: ls: not found > >> > In rank 0 and host= centos Do Barrier call 1. > >> > In rank 0 and host= centos Do Barrier call 2. > >> > In rank 1 and host= RAID Do Barrier call 1. > >> > In rank 1 and host= RAID Do Barrier call 2. > >> > > >> > Since it is such a simple test and problem and such a widely used MPI > >> > function, it must obviously > >> > be an installation or configuration problem. A pstack for each of the > >> > hung MPI_Barrier processes > >> > on the two machines shows this: > >> > > >> > [root@centos ~]# pstack 31666 > >> > #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > >> > #1 0x00007f5de06125eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> > #2 0x00007f5de061475a in opal_event_base_loop () from > >> > /usr/local/lib/libmpi.so.1 > >> > #3 0x00007f5de0639229 in opal_progress () from > >> /usr/local/lib/libmpi.so.1 > >> > #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > >> > /usr/local/lib/libmpi.so.1 > >> > #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > >> > /usr/local/lib/openmpi/mca_coll_tuned.so > >> > #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () > >> from > >> > /usr/local/lib/openmpi/mca_coll_tuned.so > >> > #7 0x00007f5de05941c2 in PMPI_Barrier () from > >> /usr/local/lib/libmpi.so.1 > >> > #8 0x0000000000400a43 in main () > >> > > >> > [root@RAID openmpi-1.6.5]# pstack 22167 > >> > #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6 > >> > #1 0x00007f7ee46885eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> > #2 0x00007f7ee468a75a in opal_event_base_loop () from > >> > /usr/local/lib/libmpi.so.1 > >> > #3 0x00007f7ee46af229 in opal_progress () from > >> /usr/local/lib/libmpi.so.1 > >> > #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > >> > /usr/local/lib/libmpi.so.1 > >> > #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > >> > /usr/local/lib/openmpi/mca_coll_tuned.so > >> > #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () > >> from > >> > /usr/local/lib/openmpi/mca_coll_tuned.so > >> > #7 0x00007f7ee460a1c2 in PMPI_Barrier () from > >> /usr/local/lib/libmpi.so.1 > >> > #8 0x0000000000400a43 in main () > >> > > >> > Which looks exactly the same on each machine. Any thoughts or ideas > >> would > >> > be greatly appreciated as > >> > I am stuck. > >> > > >> > Clay Kirkland > >> > -------------- next part -------------- > >> > HTML attachment scrubbed and removed > >> > > >> > ------------------------------ > >> > > >> > Message: 2 > >> > Date: Sat, 3 May 2014 06:39:20 -0700 > >> > From: Ralph Castain <r...@open-mpi.org> > >> > To: Open MPI Users <us...@open-mpi.org> > >> > Subject: Re: [OMPI users] MPI_Barrier hangs on second attempt but only > >> > when multiple hosts used. > >> > Message-ID: <3cf53d73-15d9-40bb-a2de-50ba3561a...@open-mpi.org> > >> > Content-Type: text/plain; charset="us-ascii" > >> > > >> > Hmmm...just testing on my little cluster here on two nodes, it works > >> just fine with 1.8.2: > >> > > >> > [rhc@bend001 v1.8]$ mpirun -n 2 --map-by node ./a.out > >> > In rank 0 and host= bend001 Do Barrier call 1. > >> > In rank 0 and host= bend001 Do Barrier call 2. > >> > In rank 0 and host= bend001 Do Barrier call 3. > >> > In rank 1 and host= bend002 Do Barrier call 1. > >> > In rank 1 and host= bend002 Do Barrier call 2. > >> > In rank 1 and host= bend002 Do Barrier call 3. > >> > [rhc@bend001 v1.8]$ > >> > > >> > > >> > How are you configuring OMPI? > >> > > >> > > >> > On May 2, 2014, at 2:24 PM, Clay Kirkland <clay.kirkl...@versityinc.com> > >> wrote: > >> > > >> > > I have been using MPI for many many years so I have very well > >> debugged mpi tests. I am > >> > > having trouble on either openmpi-1.4.5 or openmpi-1.6.5 versions > >> though with getting the > >> > > MPI_Barrier calls to work. It works fine when I run all processes > >> on one machine but when > >> > > I run with two or more hosts the second call to MPI_Barrier always > >> hangs. Not the first one, > >> > > but always the second one. I looked at FAQ's and such but found > >> nothing except for a comment > >> > > that MPI_Barrier problems were often problems with fire walls. Also > >> mentioned as a problem > >> > > was not having the same version of mpi on both machines. I turned > >> firewalls off and removed > >> > > and reinstalled the same version on both hosts but I still see the > >> same thing. I then installed > >> > > lam mpi on two of my machines and that works fine. I can call the > >> MPI_Barrier function when run on > >> > > one of two machines by itself many times with no hangs. Only hangs > >> if two or more hosts are involved. > >> > > These runs are all being done on CentOS release 6.4. Here is test > >> program I used. > >> > > > >> > > main (argc, argv) > >> > > int argc; > >> > > char **argv; > >> > > { > >> > > char message[20]; > >> > > char hoster[256]; > >> > > char nameis[256]; > >> > > int fd, i, j, jnp, iret, myrank, np, ranker, recker; > >> > > MPI_Comm comm; > >> > > MPI_Status status; > >> > > > >> > > MPI_Init( &argc, &argv ); > >> > > MPI_Comm_rank( MPI_COMM_WORLD, &myrank); > >> > > MPI_Comm_size( MPI_COMM_WORLD, &np); > >> > > > >> > > gethostname(hoster,256); > >> > > > >> > > printf(" In rank %d and host= %s Do Barrier call > >> 1.\n",myrank,hoster); > >> > > MPI_Barrier(MPI_COMM_WORLD); > >> > > printf(" In rank %d and host= %s Do Barrier call > >> 2.\n",myrank,hoster); > >> > > MPI_Barrier(MPI_COMM_WORLD); > >> > > printf(" In rank %d and host= %s Do Barrier call > >> 3.\n",myrank,hoster); > >> > > MPI_Barrier(MPI_COMM_WORLD); > >> > > MPI_Finalize(); > >> > > exit(0); > >> > > } > >> > > > >> > > Here are three runs of test program. First with two processes on > >> one host, then with > >> > > two processes on another host, and finally with one process on each > >> of two hosts. The > >> > > first two runs are fine but the last run hangs on the second > >> MPI_Barrier. > >> > > > >> > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos a.out > >> > > In rank 0 and host= centos Do Barrier call 1. > >> > > In rank 1 and host= centos Do Barrier call 1. > >> > > In rank 1 and host= centos Do Barrier call 2. > >> > > In rank 1 and host= centos Do Barrier call 3. > >> > > In rank 0 and host= centos Do Barrier call 2. > >> > > In rank 0 and host= centos Do Barrier call 3. > >> > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host RAID a.out > >> > > /root/.bashrc: line 14: unalias: ls: not found > >> > > In rank 0 and host= RAID Do Barrier call 1. > >> > > In rank 0 and host= RAID Do Barrier call 2. > >> > > In rank 0 and host= RAID Do Barrier call 3. > >> > > In rank 1 and host= RAID Do Barrier call 1. > >> > > In rank 1 and host= RAID Do Barrier call 2. > >> > > In rank 1 and host= RAID Do Barrier call 3. > >> > > [root@centos MPI]# /usr/local/bin/mpirun -np 2 --host centos,RAID > >> a.out > >> > > /root/.bashrc: line 14: unalias: ls: not found > >> > > In rank 0 and host= centos Do Barrier call 1. > >> > > In rank 0 and host= centos Do Barrier call 2. > >> > > In rank 1 and host= RAID Do Barrier call 1. > >> > > In rank 1 and host= RAID Do Barrier call 2. > >> > > > >> > > Since it is such a simple test and problem and such a widely used > >> MPI function, it must obviously > >> > > be an installation or configuration problem. A pstack for each of > >> the hung MPI_Barrier processes > >> > > on the two machines shows this: > >> > > > >> > > [root@centos ~]# pstack 31666 > >> > > #0 0x0000003baf0e8ee3 in __epoll_wait_nocancel () from > >> /lib64/libc.so.6 > >> > > #1 0x00007f5de06125eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> > > #2 0x00007f5de061475a in opal_event_base_loop () from > >> /usr/local/lib/libmpi.so.1 > >> > > #3 0x00007f5de0639229 in opal_progress () from > >> /usr/local/lib/libmpi.so.1 > >> > > #4 0x00007f5de0586f75 in ompi_request_default_wait_all () from > >> /usr/local/lib/libmpi.so.1 > >> > > #5 0x00007f5ddc59565e in ompi_coll_tuned_sendrecv_actual () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> > > #6 0x00007f5ddc59d8ff in ompi_coll_tuned_barrier_intra_two_procs () > >> from /usr/local/lib/openmpi/mca_coll_tuned.so > >> > > #7 0x00007f5de05941c2 in PMPI_Barrier () from > >> /usr/local/lib/libmpi.so.1 > >> > > #8 0x0000000000400a43 in main () > >> > > > >> > > [root@RAID openmpi-1.6.5]# pstack 22167 > >> > > #0 0x00000030302e8ee3 in __epoll_wait_nocancel () from > >> /lib64/libc.so.6 > >> > > #1 0x00007f7ee46885eb in epoll_dispatch () from > >> /usr/local/lib/libmpi.so.1 > >> > > #2 0x00007f7ee468a75a in opal_event_base_loop () from > >> /usr/local/lib/libmpi.so.1 > >> > > #3 0x00007f7ee46af229 in opal_progress () from > >> /usr/local/lib/libmpi.so.1 > >> > > #4 0x00007f7ee45fcf75 in ompi_request_default_wait_all () from > >> /usr/local/lib/libmpi.so.1 > >> > > #5 0x00007f7ee060b65e in ompi_coll_tuned_sendrecv_actual () from > >> /usr/local/lib/openmpi/mca_coll_tuned.so > >> > > #6 0x00007f7ee06138ff in ompi_coll_tuned_barrier_intra_two_procs () > >> from /usr/local/lib/openmpi/mca_coll_tuned.so > >> > > #7 0x00007f7ee460a1c2 in PMPI_Barrier () from > >> /usr/local/lib/libmpi.so.1 > >> > > #8 0x0000000000400a43 in main () > >> > > > >> > > Which looks exactly the same on each machine. Any thoughts or ideas > >> would be greatly appreciated as > >> > > I am stuck. > >> > > > >> > > Clay Kirkland > >> > > > >> > > > >> > > > >> > > > >> > > _______________________________________________ > >> > > users mailing list > >> > > us...@open-mpi.org > >> > > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > > >> > -------------- next part -------------- > >> > HTML attachment scrubbed and removed > >> > > >> > ------------------------------ > >> > > >> > Subject: Digest Footer > >> > > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > > >> > ------------------------------ > >> > > >> > End of users Digest, Vol 2879, Issue 1 > >> > ************************************** > >> > > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> > >> > >> ------------------------------ > >> > >> Subject: Digest Footer > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> ------------------------------ > >> > >> End of users Digest, Vol 2881, Issue 1 > >> ************************************** > >> > > > > > -------------- next part -------------- > HTML attachment scrubbed and removed > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > ------------------------------ > > End of users Digest, Vol 2881, Issue 2 > ************************************** > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
