Here's a core I'm getting...
[humphrey@zelda01 humphrey]$ mpiexec --mca btl_tcp_if_include eth0 --mca
oob_tcp_include eth0 -np 2 a.out
mpiexec noticed that job rank 1 with PID 20028 on node "localhost" exited on
signal 11.
1 process killed (possibly by Open MPI)
[humphrey@zelda01 humphrey]$ gdb a.out core.20028
GNU gdb Red Hat Linux (6.3.0.0-1.62rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db
library "/lib/tls/libthread_db.so.1".
Core was generated by `a.out'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /home/humphrey/ompi-install/lib/libmpi.so.0...done.
Loaded symbols for /home/humphrey/ompi-install/lib/libmpi.so.0
Reading symbols from /home/humphrey/ompi-install/lib/liborte.so.0...done.
Loaded symbols for /home/humphrey/ompi-install/lib/liborte.so.0
Reading symbols from /home/humphrey/ompi-install/lib/libopal.so.0...done.
Loaded symbols for /home/humphrey/ompi-install/lib/libopal.so.0
Reading symbols from /lib/libutil.so.1...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libaio.so.1...done.
Loaded symbols for /usr/lib/libaio.so.1
Reading symbols from /usr/lib/libg2c.so.0...done.
Loaded symbols for /usr/lib/libg2c.so.0
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_paffinity_linux.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_paffinity_linux.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ns_proxy.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ns_proxy.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ns_replica.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ns_replica.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rml_oob.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_rml_oob.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_oob_tcp.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_oob_tcp.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_gpr_null.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_gpr_null.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_gpr_proxy.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_gpr_proxy.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_gpr_replica.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_gpr_replica.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rmgr_proxy.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_rmgr_proxy.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rmgr_urm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_rmgr_urm.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rds_hostfile.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_rds_hostfile.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rds_resfile.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_rds_resfile.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ras_dash_host.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_ras_dash_host.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ras_hostfile.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_ras_hostfile.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ras_localhost.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_ras_localhost.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ras_slurm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ras_slurm.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rmaps_round_robin.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_rmaps_round_robin.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_pls_fork.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_pls_fork.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_pls_proxy.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_pls_proxy.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_pls_rsh.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_pls_rsh.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_pls_slurm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_pls_slurm.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_iof_proxy.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_iof_proxy.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_allocator_basic.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_allocator_basic.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_allocator_bucket.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_allocator_bucket.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_rcache_rb.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_rcache_rb.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_mpool_sm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_mpool_sm.so
Reading symbols from
/home/humphrey/ompi-install/lib/libmca_common_sm.so.0...done.
Loaded symbols for /home/humphrey/ompi-install/lib/libmca_common_sm.so.0
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_pml_ob1.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_pml_ob1.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_bml_r2.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_bml_r2.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_btl_self.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_btl_self.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_btl_sm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_btl_sm.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_btl_tcp.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_btl_tcp.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ptl_self.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ptl_self.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ptl_sm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ptl_sm.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_ptl_tcp.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_ptl_tcp.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_coll_basic.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_coll_basic.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_coll_hierarch.so...done.
Loaded symbols for
/home/humphrey/ompi-install/lib/openmpi/mca_coll_hierarch.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_coll_self.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_coll_self.so
Reading symbols from
/home/humphrey/ompi-install/lib/openmpi/mca_coll_sm.so...done.
Loaded symbols for /home/humphrey/ompi-install/lib/openmpi/mca_coll_sm.so
#0 0x009c4cbd in mca_btl_sm_add_procs_same_base_addr (btl=0x9c97c0,
nprocs=2, procs=0x8c28628, peers=0x8c28660,
reachability=0xbfffde80) at btl_sm.c:412
412 mca_btl_sm_component.sm_ctl_header->segment_header.
(gdb) bt
#0 0x009c4cbd in mca_btl_sm_add_procs_same_base_addr (btl=0x9c97c0,
nprocs=2, procs=0x8c28628, peers=0x8c28660,
reachability=0xbfffde80) at btl_sm.c:412
#1 0x005e7245 in mca_bml_r2_add_procs (nprocs=2, procs=0x8c28628,
bml_endpoints=0x8c28608, reachable=0xbfffde80) at bml_r2.c:220
#2 0x00323671 in mca_pml_ob1_add_procs (procs=0x8c285f8, nprocs=2) at
pml_ob1.c:131
#3 0x00ed6e81 in ompi_mpi_init (argc=0, argv=0x0, requested=0,
provided=0xbfffdf2c) at runtime/ompi_mpi_init.c:396
#4 0x00f00c62 in PMPI_Init (argc=0xbfffdf60, argv=0xbfffdf5c) at pinit.c:71
#5 0x00f2b23b in mpi_init_f (ierr=0x8052580) at pinit_f.c:65
#6 0x08049362 in MAIN__ () at Halo.f:19
#7 0x0804b7e6 in main ()
(gdb) up
#1 0x005e7245 in mca_bml_r2_add_procs (nprocs=2, procs=0x8c28628,
bml_endpoints=0x8c28608, reachable=0xbfffde80) at bml_r2.c:220
220 rc = btl->btl_add_procs(btl, n_new_procs, new_procs,
btl_endpoints, reachable);
(gdb) up
#2 0x00323671 in mca_pml_ob1_add_procs (procs=0x8c285f8, nprocs=2) at
pml_ob1.c:131
131 rc = mca_bml.bml_add_procs(
(gdb) up
#3 0x00ed6e81 in ompi_mpi_init (argc=0, argv=0x0, requested=0,
provided=0xbfffdf2c) at runtime/ompi_mpi_init.c:396
396 ret = MCA_PML_CALL(add_procs(procs, nprocs));
(gdb) up
#4 0x00f00c62 in PMPI_Init (argc=0xbfffdf60, argv=0xbfffdf5c) at pinit.c:71
71 err = ompi_mpi_init(*argc, *argv, required, &provided);
(gdb) up
#5 0x00f2b23b in mpi_init_f (ierr=0x8052580) at pinit_f.c:65
65 *ierr = OMPI_INT_2_FINT(MPI_Init( &argc, &argv ));
(gdb) up
#6 0x08049362 in MAIN__ () at Halo.f:19
19 CALL MPI_INIT(MPIERR)
Current language: auto; currently fortran
(gdb)
> -----Original Message-----
> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
> Behalf Of Jeff Squyres
> Sent: Wednesday, November 09, 2005 10:41 PM
> To: Open MPI Users
> Subject: Re: [O-MPI users] can't get openmpi to run across twomulti-
> NICmachines
>
> Sorry for the delay in replying -- it's a crazy week here preparing for
> SC next week.
>
> I'm double checking the code, and I don't see any obvious problems with
> the btl tcp include stuff.
>
> Can you also specify that you want OMPI's "out of band" communication
> to use a specific network?
>
> > mpiexec -d --mca btl_tcp_if_include eth0 --mca oob_tcp_include eth0
> > -np 2 a.out
>
> With the segv's, do you get meaningful core dumps? Can you send
> backtraces?
>
>
>
> On Nov 8, 2005, at 3:02 PM, Marty Humphrey wrote:
>
> > It's taken me a while, but I've simplified the experiment...
> >
> > In a nutshell, I'm seeing strange behavior in my multi-NIC box when I
> > attempt to execute " mpiexec -d --mca btl_tcp_if_include eth0 -np 2
> > a.out".
> > I have three different observed behaviors:
> >
> > [1] mpi worker rank 0 displays the banner and then just hangs
> > (apparently
> > trying to exchange MPI messages, which don't get delivered)
> >
> > 2 PE'S AS A 2 BY 1 GRID
> >
> > [2] it starts progressing (spitting out domain-specific msgs):
> >
> > 2 PE'S AS A 2 BY 1 GRID
> >
> > HALO2A NPES,N = 2 2 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 4 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 8 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 16 TIME = 0.000008 SECONDS
> > HALO2A NPES,N = 2 32 TIME = 0.000009 SECONDS
> >
> > [3] I get failure pretty quickly, with the line " mpiexec noticed that
> > job
> > rank 1 with PID 20425 on node "localhost" exited on signal 11."
> >
> > Here's the output of "ifconfig":
> >
> > [humphrey@zelda01 humphrey]$ /sbin/ifconfig
> > eth0 Link encap:Ethernet HWaddr 00:11:43:DC:EA:EE
> > inet addr:130.207.252.131 Bcast:130.207.252.255
> > Mask:255.255.255.0
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2441905 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:112786 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:197322445 (188.1 Mb) TX bytes:32906750 (31.3 Mb)
> > Base address:0xecc0 Memory:dfae0000-dfb00000
> >
> > eth2 Link encap:Ethernet HWaddr 00:11:95:C7:28:82
> > inet addr:10.0.0.11 Bcast:10.0.0.255 Mask:255.255.255.0
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:11598757 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:7224590 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:3491651158 (3329.8 Mb) TX bytes:1916674000 (1827.8
> > Mb)
> > Interrupt:77 Base address:0xcc00
> >
> > ipsec0 Link encap:Ethernet HWaddr 00:11:43:DC:EA:EE
> > inet addr:130.207.252.131 Mask:255.255.255.0
> > UP RUNNING NOARP MTU:16260 Metric:1
> > RX packets:40113 errors:0 dropped:40113 overruns:0 frame:0
> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:10
> > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
> >
> > lo Link encap:Local Loopback
> > inet addr:127.0.0.1 Mask:255.0.0.0
> > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > RX packets:4742 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:4742 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:2369841 (2.2 Mb) TX bytes:2369841 (2.2 Mb)
> >
> > This is with openmpi-1.1a1r8038 .
> >
> > Here is the output of a hanging invocation....
> >
> > ----- begin hanging invocation ----
> > [humphrey@zelda01 humphrey]$ mpiexec -d --mca btl_tcp_if_include eth0
> > -np 2
> > a.out
> > [zelda01.localdomain:20455] procdir: (null)
> > [zelda01.localdomain:20455] jobdir: (null)
> > [zelda01.localdomain:20455] unidir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > [zelda01.localdomain:20455] top:
> > openmpi-sessions-humphrey@zelda01.localdomain_0
> > [zelda01.localdomain:20455] tmp: /tmp
> > [zelda01.localdomain:20455] connect_uni: contact info read
> > [zelda01.localdomain:20455] connect_uni: connection not allowed
> > [zelda01.localdomain:20455] [0,0,0] setting up session dir with
> > [zelda01.localdomain:20455] tmpdir /tmp
> > [zelda01.localdomain:20455] universe default-universe-20455
> > [zelda01.localdomain:20455] user humphrey
> > [zelda01.localdomain:20455] host zelda01.localdomain
> > [zelda01.localdomain:20455] jobid 0
> > [zelda01.localdomain:20455] procid 0
> > [zelda01.localdomain:20455] procdir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20455/
> > 0/0
> > [zelda01.localdomain:20455] jobdir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20455/
> > 0
> > [zelda01.localdomain:20455] unidir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20455
> > [zelda01.localdomain:20455] top:
> > openmpi-sessions-humphrey@zelda01.localdomain_0
> > [zelda01.localdomain:20455] tmp: /tmp
> > [zelda01.localdomain:20455] [0,0,0] contact_file
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20455/
> > universe-setup.txt
> > [zelda01.localdomain:20455] [0,0,0] wrote setup file
> > [zelda01.localdomain:20455] pls:rsh: local csh: 0, local bash: 1
> > [zelda01.localdomain:20455] pls:rsh: assuming same remote shell as
> > local
> > shell
> > [zelda01.localdomain:20455] pls:rsh: remote csh: 0, remote bash: 1
> > [zelda01.localdomain:20455] pls:rsh: final template argv:
> > [zelda01.localdomain:20455] pls:rsh: ssh <template> orted --debug
> > --bootproxy 1 --name <template> --num_procs 2 --vpid_start 0 --nodename
> > <template> --universe
> > humphrey@zelda01.localdomain:default-universe-20455
> > --nsreplica
> > "0.0.0;tcp://130.207.252.131:35465;tcp://10.0.0.11:35465;tcp://
> > 130.207.252.1
> > 31:35465" --gprreplica
> > "0.0.0;tcp://130.207.252.131:35465;tcp://10.0.0.11:35465;tcp://
> > 130.207.252.1
> > 31:35465" --mpi-call-yield 0
> > [zelda01.localdomain:20455] pls:rsh: launching on node localhost
> > [zelda01.localdomain:20455] pls:rsh: oversubscribed -- setting
> > mpi_yield_when_idle to 1 (1 2)
> > [zelda01.localdomain:20455] pls:rsh: localhost is a LOCAL node
> > [zelda01.localdomain:20455] pls:rsh: executing: orted --debug
> > --bootproxy 1
> > --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename localhost
> > --universe
> > humphrey@zelda01.localdomain:default-universe-20455 --nsreplica
> > "0.0.0;tcp://130.207.252.131:35465;tcp://10.0.0.11:35465;tcp://
> > 130.207.252.1
> > 31:35465" --gprreplica
> > "0.0.0;tcp://130.207.252.131:35465;tcp://10.0.0.11:35465;tcp://
> > 130.207.252.1
> > 31:35465" --mpi-call-yield 1
> > [zelda01.localdomain:20456] [0,0,1] setting up session dir with
> > [zelda01.localdomain:20456] universe default-universe-20455
> > [zelda01.localdomain:20456] user humphrey
> > [zelda01.localdomain:20456] host localhost
> > [zelda01.localdomain:20456] jobid 0
> > [zelda01.localdomain:20456] procid 1
> > [zelda01.localdomain:20456] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/0/1
> > [zelda01.localdomain:20456] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/0
> > [zelda01.localdomain:20456] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455
> > [zelda01.localdomain:20456] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20456] tmp: /tmp
> > [zelda01.localdomain:20457] [0,1,1] setting up session dir with
> > [zelda01.localdomain:20457] universe default-universe-20455
> > [zelda01.localdomain:20457] user humphrey
> > [zelda01.localdomain:20457] host localhost
> > [zelda01.localdomain:20457] jobid 1
> > [zelda01.localdomain:20457] procid 1
> > [zelda01.localdomain:20457] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/1/1
> > [zelda01.localdomain:20457] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/1
> > [zelda01.localdomain:20457] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455
> > [zelda01.localdomain:20457] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20457] tmp: /tmp
> > [zelda01.localdomain:20458] [0,1,0] setting up session dir with
> > [zelda01.localdomain:20458] universe default-universe-20455
> > [zelda01.localdomain:20458] user humphrey
> > [zelda01.localdomain:20458] host localhost
> > [zelda01.localdomain:20458] jobid 1
> > [zelda01.localdomain:20458] procid 0
> > [zelda01.localdomain:20458] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/1/0
> > [zelda01.localdomain:20458] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455/1
> > [zelda01.localdomain:20458] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20455
> > [zelda01.localdomain:20458] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20458] tmp: /tmp
> > [zelda01.localdomain:20455] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0x3)
> > [zelda01.localdomain:20455] Info: Setting up debugger process table for
> > applications
> > MPIR_being_debugged = 0
> > MPIR_debug_gate = 0
> > MPIR_debug_state = 1
> > MPIR_acquired_pre_main = 0
> > MPIR_i_am_starter = 0
> > MPIR_proctable_size = 2
> > MPIR_proctable:
> > (i, host, exe, pid) = (0, localhost, /home/humphrey/a.out, 20457)
> > (i, host, exe, pid) = (1, localhost, /home/humphrey/a.out, 20458)
> > [zelda01.localdomain:20455] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0x4)
> > [zelda01.localdomain:20458] [0,1,0] ompi_mpi_init completed
> > [zelda01.localdomain:20457] [0,1,1] ompi_mpi_init completed
> >
> > 2 PE'S AS A 2 BY 1 GRID
> > ------ end hanging invocation -----
> >
> > Here's the 1-in-approximately-20 that started working...
> >
> > ------- begin non-hanging invocation -----
> > [humphrey@zelda01 humphrey]$ mpiexec -d --mca btl_tcp_if_include eth0
> > -np 2
> > a.out
> > [zelda01.localdomain:20659] procdir: (null)
> > [zelda01.localdomain:20659] jobdir: (null)
> > [zelda01.localdomain:20659] unidir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > [zelda01.localdomain:20659] top:
> > openmpi-sessions-humphrey@zelda01.localdomain_0
> > [zelda01.localdomain:20659] tmp: /tmp
> > [zelda01.localdomain:20659] connect_uni: contact info read
> > [zelda01.localdomain:20659] connect_uni: connection not allowed
> > [zelda01.localdomain:20659] [0,0,0] setting up session dir with
> > [zelda01.localdomain:20659] tmpdir /tmp
> > [zelda01.localdomain:20659] universe default-universe-20659
> > [zelda01.localdomain:20659] user humphrey
> > [zelda01.localdomain:20659] host zelda01.localdomain
> > [zelda01.localdomain:20659] jobid 0
> > [zelda01.localdomain:20659] procid 0
> > [zelda01.localdomain:20659] procdir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20659/
> > 0/0
> > [zelda01.localdomain:20659] jobdir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20659/
> > 0
> > [zelda01.localdomain:20659] unidir:
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20659
> > [zelda01.localdomain:20659] top:
> > openmpi-sessions-humphrey@zelda01.localdomain_0
> > [zelda01.localdomain:20659] tmp: /tmp
> > [zelda01.localdomain:20659] [0,0,0] contact_file
> > /tmp/openmpi-sessions-humphrey@zelda01.localdomain_0/default-universe
> > -20659/
> > universe-setup.txt
> > [zelda01.localdomain:20659] [0,0,0] wrote setup file
> > [zelda01.localdomain:20659] pls:rsh: local csh: 0, local bash: 1
> > [zelda01.localdomain:20659] pls:rsh: assuming same remote shell as
> > local
> > shell
> > [zelda01.localdomain:20659] pls:rsh: remote csh: 0, remote bash: 1
> > [zelda01.localdomain:20659] pls:rsh: final template argv:
> > [zelda01.localdomain:20659] pls:rsh: ssh <template> orted --debug
> > --bootproxy 1 --name <template> --num_procs 2 --vpid_start 0 --nodename
> > <template> --universe
> > humphrey@zelda01.localdomain:default-universe-20659
> > --nsreplica
> > "0.0.0;tcp://130.207.252.131:35654;tcp://10.0.0.11:35654;tcp://
> > 130.207.252.1
> > 31:35654" --gprreplica
> > "0.0.0;tcp://130.207.252.131:35654;tcp://10.0.0.11:35654;tcp://
> > 130.207.252.1
> > 31:35654" --mpi-call-yield 0
> > [zelda01.localdomain:20659] pls:rsh: launching on node localhost
> > [zelda01.localdomain:20659] pls:rsh: oversubscribed -- setting
> > mpi_yield_when_idle to 1 (1 2)
> > [zelda01.localdomain:20659] pls:rsh: localhost is a LOCAL node
> > [zelda01.localdomain:20659] pls:rsh: executing: orted --debug
> > --bootproxy 1
> > --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename localhost
> > --universe
> > humphrey@zelda01.localdomain:default-universe-20659 --nsreplica
> > "0.0.0;tcp://130.207.252.131:35654;tcp://10.0.0.11:35654;tcp://
> > 130.207.252.1
> > 31:35654" --gprreplica
> > "0.0.0;tcp://130.207.252.131:35654;tcp://10.0.0.11:35654;tcp://
> > 130.207.252.1
> > 31:35654" --mpi-call-yield 1
> > [zelda01.localdomain:20660] [0,0,1] setting up session dir with
> > [zelda01.localdomain:20660] universe default-universe-20659
> > [zelda01.localdomain:20660] user humphrey
> > [zelda01.localdomain:20660] host localhost
> > [zelda01.localdomain:20660] jobid 0
> > [zelda01.localdomain:20660] procid 1
> > [zelda01.localdomain:20660] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/0/1
> > [zelda01.localdomain:20660] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/0
> > [zelda01.localdomain:20660] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659
> > [zelda01.localdomain:20660] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20660] tmp: /tmp
> > [zelda01.localdomain:20661] [0,1,1] setting up session dir with
> > [zelda01.localdomain:20661] universe default-universe-20659
> > [zelda01.localdomain:20661] user humphrey
> > [zelda01.localdomain:20661] host localhost
> > [zelda01.localdomain:20661] jobid 1
> > [zelda01.localdomain:20661] procid 1
> > [zelda01.localdomain:20661] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/1/1
> > [zelda01.localdomain:20661] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/1
> > [zelda01.localdomain:20661] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659
> > [zelda01.localdomain:20661] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20661] tmp: /tmp
> > [zelda01.localdomain:20662] [0,1,0] setting up session dir with
> > [zelda01.localdomain:20662] universe default-universe-20659
> > [zelda01.localdomain:20662] user humphrey
> > [zelda01.localdomain:20662] host localhost
> > [zelda01.localdomain:20662] jobid 1
> > [zelda01.localdomain:20662] procid 0
> > [zelda01.localdomain:20662] procdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/1/0
> > [zelda01.localdomain:20662] jobdir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659/1
> > [zelda01.localdomain:20662] unidir:
> > /tmp/openmpi-sessions-humphrey@localhost_0/default-universe-20659
> > [zelda01.localdomain:20662] top: openmpi-sessions-humphrey@localhost_0
> > [zelda01.localdomain:20662] tmp: /tmp
> > [zelda01.localdomain:20659] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0x3)
> > [zelda01.localdomain:20659] Info: Setting up debugger process table for
> > applications
> > MPIR_being_debugged = 0
> > MPIR_debug_gate = 0
> > MPIR_debug_state = 1
> > MPIR_acquired_pre_main = 0
> > MPIR_i_am_starter = 0
> > MPIR_proctable_size = 2
> > MPIR_proctable:
> > (i, host, exe, pid) = (0, localhost, /home/humphrey/a.out, 20661)
> > (i, host, exe, pid) = (1, localhost, /home/humphrey/a.out, 20662)
> > [zelda01.localdomain:20659] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0x4)
> > [zelda01.localdomain:20662] [0,1,0] ompi_mpi_init completed
> > [zelda01.localdomain:20661] [0,1,1] ompi_mpi_init completed
> >
> > 2 PE'S AS A 2 BY 1 GRID
> >
> > HALO2A NPES,N = 2 2 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 4 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 8 TIME = 0.000007 SECONDS
> > HALO2A NPES,N = 2 16 TIME = 0.000008 SECONDS
> > HALO2A NPES,N = 2 32 TIME = 0.000009 SECONDS
> > HALO2A NPES,N = 2 64 TIME = 0.000011 SECONDS
> > mpiexec: killing job...
> > Interrupt
> > Interrupt
> > [zelda01.localdomain:20660] sess_dir_finalize: found proc session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: job session dir not
> > empty -
> > leaving
> > [zelda01.localdomain:20660] sess_dir_finalize: found proc session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: found job session dir
> > empty -
> > deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: univ session dir not
> > empty -
> > leaving
> > [zelda01.localdomain:20659] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0xa)
> > [zelda01.localdomain:20660] orted: job_state_callback(jobid = 1, state
> > =
> > ORTE_PROC_STATE_ABORTED)
> > [zelda01.localdomain:20659] spawn: in job_state_callback(jobid = 1,
> > state =
> > 0x9)
> > 2 processes killed (possibly by Open MPI)
> > [zelda01.localdomain:20660] orted: job_state_callback(jobid = 1, state
> > =
> > ORTE_PROC_STATE_TERMINATED)
> > [zelda01.localdomain:20660] sess_dir_finalize: found proc session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: found job session dir
> > empty -
> > deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: found univ session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20660] sess_dir_finalize: found top session dir
> > empty -
> > deleting
> > [zelda01.localdomain:20659] sess_dir_finalize: found proc session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20659] sess_dir_finalize: found job session dir
> > empty -
> > deleting
> > [zelda01.localdomain:20659] sess_dir_finalize: found univ session dir
> > empty
> > - deleting
> > [zelda01.localdomain:20659] sess_dir_finalize: top session dir not
> > empty -
> > leaving
> > [humphrey@zelda01 humphrey]$
> > -------- end non-hanging invocation ------
> >
> > Any thoughts?
> >
> > -- Marty
> >
> >> -----Original Message-----
> >> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org]
> >> On
> >> Behalf Of Jeff Squyres
> >> Sent: Tuesday, November 01, 2005 2:17 PM
> >> To: Open MPI Users
> >> Subject: Re: [O-MPI users] can't get openmpi to run across two multi-
> >> NICmachines
> >>
> >> On Nov 1, 2005, at 12:02 PM, Marty Humphrey wrote:
> >>
> >>> wukong: eth0 (152.48.249.102, no MPI traffic), eth1
> >>> (128.109.34.20,yes
> >>> MPI
> >>> traffic)
> >>> zelda01: eth0 (130.207.252.131, yes MPI traffic), eth2 (10.0.0.12, no
> >>> MPI
> >>> traffic)
> >>>
> >>> on wukong, I have :
> >>> [humphrey@wukong ~]$ more ~/.openmpi/mca-params.conf
> >>> btl_tcp_if_include=eth1
> >>> on zelda01, I have :
> >>> [humphrey@zelda01 humphrey]$ more ~/.openmpi/mca-params.conf
> >>> btl_tcp_if_include=eth0
> >>
> >> Just to make sure I'm reading this right -- 128.109.34.20 is supposed
> >> to be routable to 130.207.252.131, right? Can you ssh directly from
> >> one machine to the other? (I'm guessing that you can because OMPI was
> >> able to start processes) Can you ping one machine from the other?
> >>
> >> Most importantly -- can you open arbitrary TCP ports between the two
> >> machines? (i.e., not just well-known ports like 22 [ssh], etc.)
> >>
> >> --
> >> {+} Jeff Squyres
> >> {+} The Open MPI Project
> >> {+} http://www.open-mpi.org/
> >>
> >> _______________________________________________
> >> users mailing list
> >> us...@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
> --
> {+} Jeff Squyres
> {+} The Open MPI Project
> {+} http://www.open-mpi.org/
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
