That works!! Thanks!! George Bosilca wrote: >Sorry I wasn't clear enough on my previous post. The error messages that >you get are comming from the OOB which is the framework we're using to >setup the MPI run. The options that you use (btl_tcp_if_include) are only >used for MPI communications. Please add "--mca oob_tcp_include eth0" to >force the OOB framework to use eth0. In order to don't have to type all >these options all the time you can add them in the >$(HOME).openmpi/mca-params.conf file. A file containing: > >oob_tcp_include=eth1 >btl_tcp_if_include=eth1 > >should solve your problems, if the firewall is opened on eth1 between >these nodes. > > Thanks, > george. > >On Thu, 16 Mar 2006, Charles Wright wrote: > > >>Thanks for the tip. >> >>I see that both number 1 and 2 are true. >>Openmpi is insisting on using my eth0 (I know this by watching the >>firewall log on the node it is trying to go to) >> >>This is despite the fact that I have the first dns entry go to eth1, >>normally that is all pbs would need to do the right thing and use the >>network I prefer. >> >>Ok so I see there are some options to in/exclude interfaces. >> >>however mpiexec is igorning my requests. >>I tried it two ways. Neither worked. Firewall rejects traffic coming >>into 1.0.x.x. network in both cases. >> >>/opt/asn/apps/openmpi-1.0.1/bin/mpiexec --gmca btl_tcp_if_include eth1 >>-n 2 $XD1LAUNCHER ./mpimeasure >>/opt/asn/apps/openmpi-1.0.1/bin/mpiexec --gmca btl_tcp_if_exclude eth0 >>-n 2 $XD1LAUNCHER ./mpimeasure >> >>(see dns works... not over eth0) >>uahrcw@c344-6:~/mpi-benchmarks> /sbin/ifconfig >>eth0 Link encap:Ethernet HWaddr 00:0E:AB:01:58:60 >> inet addr:1.0.21.134 Bcast:1.127.255.255 Mask:255.128.0.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:6596091 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:316165 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:560395541 (534.4 Mb) TX bytes:34367848 (32.7 Mb) >> Interrupt:16 >> >>eth1 Link encap:Ethernet HWaddr 00:0E:AB:01:58:61 >> inet addr:1.128.21.134 Mask:255.128.0.0 >> UP RUNNING NOARP MTU:1500 Metric:1 >> RX packets:5600487 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:4863441 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:6203028277 (5915.6 Mb) TX bytes:566471561 (540.2 Mb) >> Interrupt:25 >> >>eth2 Link encap:Ethernet HWaddr 00:0E:AB:01:58:62 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:829064 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:181572 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:61216408 (58.3 Mb) TX bytes:19079579 (18.1 Mb) >> Base address:0x2000 Memory:fea80000-feaa0000 >> >>eth2:2 Link encap:Ethernet HWaddr 00:0E:AB:01:58:62 >> inet addr:129.66.9.146 Bcast:129.66.9.255 Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> Base address:0x2000 Memory:fea80000-feaa0000 >> >>lo Link encap:Local Loopback >> inet addr:127.0.0.1 Mask:255.0.0.0 >> UP LOOPBACK RUNNING MTU:16436 Metric:1 >> RX packets:14259 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:14259 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:879631 (859.0 Kb) TX bytes:879631 (859.0 Kb) >> >>uahrcw@c344-6:~/mpi-benchmarks> ping c344-5 >>PING c344-5.x.asc.edu (1.128.21.133) 56(84) bytes of data. >>64 bytes from c344-5.x.asc.edu (1.128.21.133): icmp_seq=1 ttl=64 >>time=0.067 ms >>64 bytes from c344-5.x.asc.edu (1.128.21.133): icmp_seq=2 ttl=64 >>time=0.037 ms >>64 bytes from c344-5.x.asc.edu (1.128.21.133): icmp_seq=3 ttl=64 >>time=0.022 ms >> >>--- c344-5.x.asc.edu ping statistics --- >>3 packets transmitted, 3 received, 0% packet loss, time 1999ms >>rtt min/avg/max/mdev = 0.022/0.042/0.067/0.018 ms >> >> >> >>George Bosilca wrote: >> >>>I see only 2 possibilities: >>>1. your trying to run Open MPI on nodes having multiple IP >>>addresses. >>>2. your nodes are behind firewalls and Open MPI is unable to pass through. >>> >>>Please check the FAQ on http://www.open-mpi.org/faq/ to find out the full >>>answer to your question. >>> >>> Thanks, >>> george. >>> >>>On Thu, 16 Mar 2006, Charles Wright wrote: >>> >>> >>> >>>>Hello, >>>> I'm just compiled open-mpi and tried to run my code which just >>>>measures bandwidth from one node to another. (Code compile fine and >>>>runs under other mpi implementations) >>>> >>>>When I did I got this. >>>> >>>>uahrcw@c275-6:~/mpi-benchmarks> cat openmpitcp.o15380 >>>>c317-6 >>>>c317-5 >>>>[c317-5:24979] [0,0,2]-[0,0,0] mca_oob_tcp_peer_complete_connect: >>>>connection failed (errno=110) - retrying (pid=24979) >>>>[c317-5:24979] mca_oob_tcp_peer_timer_handler >>>>[c317-5:24997] [0,1,1]-[0,0,0] mca_oob_tcp_peer_complete_connect: >>>>connection failed (errno=110) - retrying (pid=24997) >>>>[c317-5:24997] mca_oob_tcp_peer_timer_handler >>>> >>>>[0,1,1][btl_tcp_endpoint.c:559:mca_btl_tcp_endpoint_complete_connect] >>>>connect() failed with errno=110 >>>> >>>> >>>>I compiled open-mpi with Pbspro 5.4-4 and I'm guessing that has >>>>something to do with it. >>>> >>>>I've attached my config.log >>>> >>>>Any help with this would be appreciated. >>>> >>>>uahrcw@c275-6:~/mpi-benchmarks> ompi_info >>>> Open MPI: 1.0.1r8453 >>>> Open MPI SVN revision: r8453 >>>> Open RTE: 1.0.1r8453 >>>> Open RTE SVN revision: r8453 >>>> OPAL: 1.0.1r8453 >>>> OPAL SVN revision: r8453 >>>> Prefix: /opt/asn/apps/openmpi-1.0.1 >>>>Configured architecture: x86_64-unknown-linux-gnu >>>> Configured by: asnrcw >>>> Configured on: Fri Feb 24 15:19:37 CST 2006 >>>> Configure host: c275-6 >>>> Built by: asnrcw >>>> Built on: Fri Feb 24 15:40:09 CST 2006 >>>> Built host: c275-6 >>>> C bindings: yes >>>> C++ bindings: yes >>>> Fortran77 bindings: yes (all) >>>> Fortran90 bindings: no >>>> C compiler: gcc >>>> C compiler absolute: /usr/bin/gcc >>>> C++ compiler: g++ >>>> C++ compiler absolute: /usr/bin/g++ >>>> Fortran77 compiler: g77 >>>>Fortran77 compiler abs: /usr/bin/g77 >>>> Fortran90 compiler: ifort >>>>Fortran90 compiler abs: /opt/asn/intel/fce/9.0/bin/ifort >>>> C profiling: yes >>>> C++ profiling: yes >>>> Fortran77 profiling: yes >>>> Fortran90 profiling: no >>>> C++ exceptions: no >>>> Thread support: posix (mpi: no, progress: no) >>>>Internal debug support: no >>>> MPI parameter check: runtime >>>>Memory profiling support: no >>>>Memory debugging support: no >>>> libltdl support: 1 >>>> MCA memory: malloc_hooks (MCA v1.0, API v1.0, Component >>>>v1.0.1) >>>> MCA paffinity: linux (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA maffinity: first_use (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA maffinity: libnuma (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA timer: linux (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA allocator: basic (MCA v1.0, API v1.0, Component v1.0) >>>> MCA allocator: bucket (MCA v1.0, API v1.0, Component v1.0) >>>> MCA coll: basic (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA coll: self (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA coll: sm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA io: romio (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA mpool: sm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pml: ob1 (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pml: teg (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ptl: self (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ptl: sm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ptl: tcp (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA btl: self (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA btl: sm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA btl: tcp (MCA v1.0, API v1.0, Component v1.0) >>>> MCA topo: unity (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA gpr: null (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA gpr: proxy (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA gpr: replica (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA iof: proxy (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA iof: svc (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ns: proxy (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ns: replica (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA oob: tcp (MCA v1.0, API v1.0, Component v1.0) >>>> MCA ras: dash_host (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ras: hostfile (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ras: localhost (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ras: slurm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA ras: tm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rds: hostfile (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rds: resfile (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rmaps: round_robin (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rmgr: proxy (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rmgr: urm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA rml: oob (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: daemon (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: proxy (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: fork (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: rsh (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: slurm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA pls: tm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA sds: env (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA sds: seed (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA sds: singleton (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA sds: slurm (MCA v1.0, API v1.0, Component v1.0.1) >>>> MCA sds: pipe (MCA v1.0, API v1.0, Component v1.0.1) >>>>uahrcw@c275-6:~/mpi-benchmarks> >>>> >>>> >>>> >>>> >>>> >>>"We must accept finite disappointment, but we must never lose infinite >>>hope." >>> Martin Luther King >>> >>>_______________________________________________ >>>users mailing list >>>us...@open-mpi.org >>>http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >> >> > >"We must accept finite disappointment, but we must never lose infinite >hope." > Martin Luther King > >_______________________________________________ >users mailing list >us...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/users > >
-- Charles Wright, HPC Systems Administrator Alabama Research and Education Network Computer Sciences Corporation
