An other test is to swap the hostnames. If the single barrier test fails, this can hint to a firewall.
Cheers, Gilles Gilles Gouaillardet <gil...@rist.or.jp> wrote: >sudo make uninstall >will not remove modules that are no more built >sudo rm -rf /usr/local/lib/openmpi >is safe thought > >i confirm i did not see any issue on a system with two networks > >Cheers, > >Gilles > >On 4/18/2016 2:53 PM, dpchoudh . wrote: > >Hello Gilles > >I did a > >sudo make uninstall > >followed by a > >sudo make install > >on both nodes. But that did not make a difference. I will try your tarball >build suggestion a bit later. > >What I find a bit strange is that only I seem to be getting into this issue. >What could I be doing wrong? Or am I discovering an obscure bug? > >Thanks > >Durga > > >1% of the executables have 99% of CPU privilege! > >Userspace code! Unite!! Occupy the kernel!!! > > >On Mon, Apr 18, 2016 at 1:21 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote: > >so you might want to >rm -rf /usr/local/lib/openmpi >and run >make install >again, just to make sure old stuff does not get in the way > >Cheers, > >Gilles > > > >On 4/18/2016 2:12 PM, dpchoudh . wrote: > >Hello Gilles > >Thank you very much for your feedback. You are right that my original stack >trace was on code that was several weeks behind, but updating it just now did >not seem to make a difference: I am copying the stack from the latest code >below: > > >On the master node: > >(gdb) bt >#0 0x00007fc0524cbb7d in poll () from /lib64/libc.so.6 >#1 0x00007fc051e53116 in poll_dispatch (base=0x1aabbe0, tv=0x7fff29fcb240) at >poll.c:165 >#2 0x00007fc051e4adb0 in opal_libevent2022_event_base_loop (base=0x1aabbe0, >flags=2) at event.c:1630 >#3 0x00007fc051de9a00 in opal_progress () at runtime/opal_progress.c:171 >#4 0x00007fc04ce46b0b in opal_condition_wait (c=0x7fc052d3cde0 ><ompi_request_cond>, > m=0x7fc052d3cd60 <ompi_request_lock>) at >../../../../opal/threads/condition.h:76 >#5 0x00007fc04ce46cec in ompi_request_wait_completion (req=0x1b7b580) > at ../../../../ompi/request/request.h:383 >#6 0x00007fc04ce48d4f in mca_pml_ob1_send (buf=0x7fff29fcb480, count=4, > datatype=0x601080 <ompi_mpi_char>, dst=1, tag=1, >sendmode=MCA_PML_BASE_SEND_STANDARD, > comm=0x601280 <ompi_mpi_comm_world>) at pml_ob1_isend.c:259 >#7 0x00007fc052a62d73 in PMPI_Send (buf=0x7fff29fcb480, count=4, >type=0x601080 <ompi_mpi_char>, dest=1, > tag=1, comm=0x601280 <ompi_mpi_comm_world>) at psend.c:78 >#8 0x0000000000400afa in main (argc=1, argv=0x7fff29fcb5e8) at mpitest.c:19 >(gdb) > >And on the non-master node > >(gdb) bt >#0 0x00007fad2c32148d in nanosleep () from /lib64/libc.so.6 >#1 0x00007fad2c352014 in usleep () from /lib64/libc.so.6 >#2 0x00007fad296412de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, nprocs=0, >info=0x0, ninfo=0) > at src/client/pmix_client_fence.c:100 >#3 0x00007fad2960e1a6 in pmix120_fence (procs=0x0, collect_data=0) at >pmix120_client.c:258 >#4 0x00007fad2c89b2da in ompi_mpi_finalize () at >runtime/ompi_mpi_finalize.c:242 >#5 0x00007fad2c8c5849 in PMPI_Finalize () at pfinalize.c:47 >#6 0x0000000000400958 in main (argc=1, argv=0x7fff163879c8) at mpitest.c:30 >(gdb) > >And my configuration was done as follows: > > $ ./configure --enable-debug --enable-debug-symbols > >I double checked to ensure that there is not an older installation of OpenMPI >that is getting mixed up with the master branch. > >sudo yum list installed | grep -i mpi > >shows nothing on both nodes, and pmap -p <pid> shows that all the libraries >are coming from /usr/local/lib, which seems to be correct. I am also quite >sure about the firewall issue (that there is none). I will try out your >suggestion on installing from a tarball and see how it goes. > > >Thanks > >Durga > > >1% of the executables have 99% of CPU privilege! > >Userspace code! Unite!! Occupy the kernel!!! > > >On Mon, Apr 18, 2016 at 12:47 AM, Gilles Gouaillardet <gil...@rist.or.jp> >wrote: > >here is your stack trace > >#6 0x00007f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4, > datatype=0x601080 <ompi_mpi_char>, dst=1, tag=1, > sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280 <ompi_mpi_comm_world>) > >at line 251 > > >that would be line 259 in current master, and this file was updated 21 days ago >and that suggests your master is not quite up to date. > >even if the message is sent eagerly, the ob1 pml does use an internal request >it will wait for. > >btw, did you configure with --enable-mpi-thread-multiple ? >did you configure with --enable-mpirun-prefix-by-default ? >did you configure with --disable-dlopen ? > >at first, i d recommend you download a tarball from >https://www.open-mpi.org/nightly/master, >configure && make && make install >using a new install dir, and check if the issue is still here or not. > >there could be some side effects if some old modules were not removed and/or >if you are >not using the modules you expect. >/* when it hangs, you can pmap <pid> and check the path of the openmpi >libraries are the one you expect */ > >what if you do not send/recv but invoke MPI_Barrier multiple times ? >what if you send/recv a one byte message instead ? >did you double check there is no firewall running on your nodes ? > >Cheers, > >Gilles > > > > > > > >On 4/18/2016 1:06 PM, dpchoudh . wrote: > >Thank you for your suggestion, Ralph. But it did not make any difference. > >Let me say that my code is about a week stale. I just did a git pull and am >building it right now. The build takes quite a bit of time, so I avoid doing >that unless there is a reason. But what I am trying out is the most basic >functionality, so I'd think a week or so of lag would not make a difference. > >Does the stack trace suggest something to you? It seems that the send hangs; >but a 4 byte send should be sent eagerly. > >Best regards > >'Durga > > >1% of the executables have 99% of CPU privilege! > >Userspace code! Unite!! Occupy the kernel!!! > > >On Sun, Apr 17, 2016 at 11:55 PM, Ralph Castain <r...@open-mpi.org> wrote: > >Try adding -mca oob_tcp_if_include eno1 to your cmd line and see if that makes >a difference > > >On Apr 17, 2016, at 8:43 PM, dpchoudh . <dpcho...@gmail.com> wrote: > > >Hello Gilles and all > >I am sorry to be bugging the developers, but this issue seems to be nagging >me, and I am surprised it does not seem to affect anybody else. But then >again, I am using the master branch, and most users are probably using a >released version. > >This time I am using a totally different cluster. This has NO verbs capable >interface; just 2 Ethernet (1 of which has no IP address and hence is >unusable) plus 1 proprietary interface that currently supports only IP >traffic. The two IP interfaces (Ethernet and proprietary) are on different IP >subnets. > >My test program is as follows: > >#include <stdio.h> >#include <string.h> >#include "mpi.h" >int main(int argc, char *argv[]) >{ >char host[128]; >int n; >MPI_Init(&argc, &argv); >MPI_Get_processor_name(host, &n); >printf("Hello from %s\n", host); >MPI_Comm_size(MPI_COMM_WORLD, &n); >printf("The world has %d nodes\n", n); >MPI_Comm_rank(MPI_COMM_WORLD, &n); >printf("My rank is %d\n",n); >//#if 0 >if (n == 0) >{ >strcpy(host, "ha!"); >MPI_Send(host, strlen(host) + 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD); >printf("sent %s\n", host); >} >else >{ >//int len = strlen(host) + 1; >bzero(host, 128); >MPI_Recv(host, 4, MPI_CHAR, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE); >printf("Received %s from rank 0\n", host); >} >//#endif >MPI_Finalize(); >return 0; >} > >This program, when run between two nodes, hangs. The command was: >[durga@b-1 ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca pml >ob1 -mca btl_tcp_if_include eno1 ./mpitest > >And the hang is with the following output: (eno1 is one of the gigEth >interfaces, that takes OOB traffic as well) > >Hello from b-1 >The world has 2 nodes >My rank is 0 >Hello from b-2 >The world has 2 nodes >My rank is 1 > >Note that if I uncomment the #if 0 - #endif (i.e. comment out the >MPI_Send()/MPI_Recv() part, the program runs to completion. Also note that the >printfs following MPI_Send()/MPI_Recv() do not show up on console. > >Upon attaching gdb, the stack trace from the master node is as follows: > >Missing separate debuginfos, use: debuginfo-install glibc-2.17-78.el7.x86_64 >libpciaccess-0.13.4-2.el7.x86_64 >(gdb) bt >#0 0x00007f72a533eb7d in poll () from /lib64/libc.so.6 >#1 0x00007f72a4cb7146 in poll_dispatch (base=0xee33d0, tv=0x7fff81057b70) > at poll.c:165 >#2 0x00007f72a4caede0 in opal_libevent2022_event_base_loop (base=0xee33d0, > flags=2) at event.c:1630 >#3 0x00007f72a4c4e692 in opal_progress () at runtime/opal_progress.c:171 >#4 0x00007f72a0d07ac1 in opal_condition_wait ( > c=0x7f72a5bb1e00 <ompi_request_cond>, m=0x7f72a5bb1d80 <ompi_request_lock>) > at ../../../../opal/threads/condition.h:76 >#5 0x00007f72a0d07ca2 in ompi_request_wait_completion (req=0x113eb80) > at ../../../../ompi/request/request.h:383 >#6 0x00007f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0, count=4, > datatype=0x601080 <ompi_mpi_char>, dst=1, tag=1, > sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280 <ompi_mpi_comm_world>) > at pml_ob1_isend.c:251 >#7 0x00007f72a58d6be3 in PMPI_Send (buf=0x7fff81057db0, count=4, > type=0x601080 <ompi_mpi_char>, dest=1, tag=1, > comm=0x601280 <ompi_mpi_comm_world>) at psend.c:78 >#8 0x0000000000400afa in main (argc=1, argv=0x7fff81057f18) at mpitest.c:19 >(gdb) > >And the backtrace on the non-master node is: > >(gdb) bt >#0 0x00007ff3b377e48d in nanosleep () from /lib64/libc.so.6 >#1 0x00007ff3b37af014 in usleep () from /lib64/libc.so.6 >#2 0x00007ff3b0c922de in OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, nprocs=0, > info=0x0, ninfo=0) at src/client/pmix_client_fence.c:100 >#3 0x00007ff3b0c5f1a6 in pmix120_fence (procs=0x0, collect_data=0) > at pmix120_client.c:258 >#4 0x00007ff3b3cf8f4b in ompi_mpi_finalize () > at runtime/ompi_mpi_finalize.c:242 >#5 0x00007ff3b3d23295 in PMPI_Finalize () at pfinalize.c:47 >#6 0x0000000000400958 in main (argc=1, argv=0x7fff785e8788) at mpitest.c:30 >(gdb) > >The hostfile is as follows: > >[durga@b-1 ~]$ cat hostfile >10.4.70.10 slots=1 >10.4.70.11 slots=1 >#10.4.70.12 slots=1 > >And the ifconfig output from the master node is as follows (the other node is >similar; all the IP interfaces are in their respective subnets) : > >[durga@b-1 ~]$ ifconfig >eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 10.4.70.10 netmask 255.255.255.0 broadcast 10.4.70.255 > inet6 fe80::21e:c9ff:fefe:13df prefixlen 64 scopeid 0x20<link> > ether 00:1e:c9:fe:13:df txqueuelen 1000 (Ethernet) > RX packets 48215 bytes 27842846 (26.5 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 52746 bytes 7817568 (7.4 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 16 > >eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 > ether 00:1e:c9:fe:13:e0 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 > >lf0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2016 > inet 192.168.1.2 netmask 255.255.255.0 broadcast 192.168.1.255 > inet6 fe80::3002:ff:fe33:3333 prefixlen 64 scopeid 0x20<link> > ether 32:02:00:33:33:33 txqueuelen 1000 (Ethernet) > RX packets 10 bytes 512 (512.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 22 bytes 1536 (1.5 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > >lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > inet 127.0.0.1 netmask 255.0.0.0 > inet6 ::1 prefixlen 128 scopeid 0x10<host> > loop txqueuelen 0 (Local Loopback) > RX packets 26 bytes 1378 (1.3 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 26 bytes 1378 (1.3 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > >Please help me with this. I am stuck with the TCP transport, which is the most >basic of all transports. > >Thanks in advance > >Durga > > > >1% of the executables have 99% of CPU privilege! > >Userspace code! Unite!! Occupy the kernel!!! > > >On Tue, Apr 12, 2016 at 9:32 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote: > >This is quite unlikely, and fwiw, your test program works for me. > >i suggest you check your 3 TCP networks are usable, for example > >$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca pml ob1 --mca >btl_tcp_if_include xxx ./mpitest > >in which xxx is a [list of] interface name : >eth0 >eth1 >ib0 >eth0,eth1 >eth0,ib0 >... >eth0,eth1,ib0 > >and see where problem start occuring. > >btw, are your 3 interfaces in 3 different subnet ? is routing required between >two interfaces of the same type ? > >Cheers, > >Gilles > > >On 4/13/2016 7:15 AM, dpchoudh . wrote: > >Hi all > >I have reported this issue before, but then had brushed it off as something >that was caused by my modifications to the source tree. It looks like that is >not the case. > >Just now, I did the following: > >1. Cloned a fresh copy from master. > >2. Configured with the following flags, built and installed it in my two-node >"cluster". > >--enable-debug --enable-debug-symbols --disable-dlopen > >3. Compiled the following program, mpitest.c with these flags: -g3 -Wall >-Wextra > >4. Ran it like this: >[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca >pml ob1 ./mpitest > >With this, the code hangs at MPI_Barrier() on both nodes, after generating the >following output: > >Hello world from processor smallMPI, rank 0 out of 2 processors >Hello world from processor bigMPI, rank 1 out of 2 processors >smallMPI sent haha! >bigMPI received haha! > ><Hangs until killed by ^C> > >Attaching to the hung process at one node gives the following backtrace: > >(gdb) bt >#0 0x00007f55b0f41c3d in poll () from /lib64/libc.so.6 >#1 0x00007f55b03ccde6 in poll_dispatch (base=0x70e7b0, tv=0x7ffd1bb551c0) at >poll.c:165 >#2 0x00007f55b03c4a90 in opal_libevent2022_event_base_loop (base=0x70e7b0, >flags=2) at event.c:1630 >#3 0x00007f55b02f0144 in opal_progress () at runtime/opal_progress.c:171 >#4 0x00007f55b14b4d8b in opal_condition_wait (c=0x7f55b19fec40 ><ompi_request_cond>, m=0x7f55b19febc0 <ompi_request_lock>) at >../opal/threads/condition.h:76 >#5 0x00007f55b14b531b in ompi_request_default_wait_all (count=2, >requests=0x7ffd1bb55370, statuses=0x7ffd1bb55340) at request/req_wait.c:287 >#6 0x00007f55b157a225 in ompi_coll_base_sendrecv_zero (dest=1, stag=-16, >source=1, rtag=-16, comm=0x601280 <ompi_mpi_comm_world>) > at base/coll_base_barrier.c:63 >#7 0x00007f55b157a92a in ompi_coll_base_barrier_intra_two_procs >(comm=0x601280 <ompi_mpi_comm_world>, module=0x7c2630) at >base/coll_base_barrier.c:308 >#8 0x00007f55b15aafec in ompi_coll_tuned_barrier_intra_dec_fixed >(comm=0x601280 <ompi_mpi_comm_world>, module=0x7c2630) at >coll_tuned_decision_fixed.c:196 >#9 0x00007f55b14d36fd in PMPI_Barrier (comm=0x601280 <ompi_mpi_comm_world>) >at pbarrier.c:63 >#10 0x0000000000400b0b in main (argc=1, argv=0x7ffd1bb55658) at mpitest.c:26 >(gdb) > >Thinking that this might be a bug in tuned collectives, since that is what the >stack shows, I ran the program like this (basically adding the ^tuned part) > >[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca >pml ob1 -mca coll ^tuned ./mpitest > >It still hangs, but now with a different stack trace: >(gdb) bt >#0 0x00007f910d38ac3d in poll () from /lib64/libc.so.6 >#1 0x00007f910c815de6 in poll_dispatch (base=0x1a317b0, tv=0x7fff43ee3610) at >poll.c:165 >#2 0x00007f910c80da90 in opal_libevent2022_event_base_loop (base=0x1a317b0, >flags=2) at event.c:1630 >#3 0x00007f910c739144 in opal_progress () at runtime/opal_progress.c:171 >#4 0x00007f910db130f7 in opal_condition_wait (c=0x7f910de47c40 ><ompi_request_cond>, m=0x7f910de47bc0 <ompi_request_lock>) > at ../../../../opal/threads/condition.h:76 >#5 0x00007f910db132d8 in ompi_request_wait_completion (req=0x1b07680) at >../../../../ompi/request/request.h:383 >#6 0x00007f910db1533b in mca_pml_ob1_send (buf=0x0, count=0, >datatype=0x7f910de1e340 <ompi_mpi_byte>, dst=1, tag=-16, >sendmode=MCA_PML_BASE_SEND_STANDARD, > comm=0x601280 <ompi_mpi_comm_world>) at pml_ob1_isend.c:259 >#7 0x00007f910d9c3b38 in ompi_coll_base_barrier_intra_basic_linear >(comm=0x601280 <ompi_mpi_comm_world>, module=0x1b092c0) at >base/coll_base_barrier.c:368 >#8 0x00007f910d91c6fd in PMPI_Barrier (comm=0x601280 <ompi_mpi_comm_world>) >at pbarrier.c:63 >#9 0x0000000000400b0b in main (argc=1, argv=0x7fff43ee3a58) at mpitest.c:26 >(gdb) > >The mpitest.c program is as follows: >#include <mpi.h> >#include <stdio.h> >#include <string.h> > >int main(int argc, char** argv) >{ > int world_size, world_rank, name_len; > char hostname[MPI_MAX_PROCESSOR_NAME], buf[8]; > > MPI_Init(&argc, &argv); > MPI_Comm_size(MPI_COMM_WORLD, &world_size); > MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); > MPI_Get_processor_name(hostname, &name_len); > printf("Hello world from processor %s, rank %d out of %d processors\n", >hostname, world_rank, world_size); > if (world_rank == 1) > { > MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD, MPI_STATUS_IGNORE); > printf("%s received %s\n", hostname, buf); > } > else > { > strcpy(buf, "haha!"); > MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD); > printf("%s sent %s\n", hostname, buf); > } > MPI_Barrier(MPI_COMM_WORLD); > MPI_Finalize(); > return 0; >} > >The hostfile is as follows: >10.10.10.10 slots=1 >10.10.10.11 slots=1 > >The two nodes are connected by three physical and 3 logical networks: > >Physical: Gigabit Ethernet, 10G iWARP, 20G Infiniband > >Logical: IP (all 3), PSM (Qlogic Infiniband), Verbs (iWARP and Infiniband) > >Please note again that this is a fresh, brand new clone. > >Is this a bug (perhaps a side effect of --disable-dlopen) or something I am >doing wrong? > >Thanks > >Durga > > >We learn from history that we never learn from history. > > > >_______________________________________________ users mailing list >us...@open-mpi.org Subscription: >http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28930.php > > > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28932.php > > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28942.php > > > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28943.php > > > > >_______________________________________________ users mailing list >us...@open-mpi.org Subscription: >http://www.open-mpi.org/mailman/listinfo.cgi/users > >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28944.php > > > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28946.php > > > > >_______________________________________________ users mailing list >us...@open-mpi.org Subscription: >http://www.open-mpi.org/mailman/listinfo.cgi/users > >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28947.php > > > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28948.php > > > > >_______________________________________________ users mailing list >us...@open-mpi.org Subscription: >http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: >http://www.open-mpi.org/community/lists/users/2016/04/28949.php > >
