On Apr 17, 2016, at 8:43 PM, dpchoudh .
<dpcho...@gmail.com
<mailto:dpcho...@gmail.com>> wrote:
Hello Gilles and all
I am sorry to be bugging the developers, but
this issue seems to be nagging me, and I am
surprised it does not seem to affect anybody
else. But then again, I am using the master
branch, and most users are probably using a
released version.
This time I am using a totally different
cluster. This has NO verbs capable interface;
just 2 Ethernet (1 of which has no IP address
and hence is unusable) plus 1 proprietary
interface that currently supports only IP
traffic. The two IP interfaces (Ethernet and
proprietary) are on different IP subnets.
My test program is as follows:
#include <stdio.h>
#include <string.h>
#include "mpi.h"
int main(int argc, char *argv[])
{
char host[128];
int n;
MPI_Init(&argc, &argv);
MPI_Get_processor_name(host, &n);
printf("Hello from %s\n", host);
MPI_Comm_size(MPI_COMM_WORLD, &n);
printf("The world has %d nodes\n", n);
MPI_Comm_rank(MPI_COMM_WORLD, &n);
printf("My rank is %d\n",n);
//#if 0
if (n == 0)
{
strcpy(host, "ha!");
MPI_Send(host, strlen(host) + 1, MPI_CHAR, 1,
1, MPI_COMM_WORLD);
printf("sent %s\n", host);
}
else
{
//int len = strlen(host) + 1;
bzero(host, 128);
MPI_Recv(host, 4, MPI_CHAR, 0, 1,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("Received %s from rank 0\n", host);
}
//#endif
MPI_Finalize();
return 0;
}
This program, when run between two nodes,
hangs. The command was:
[durga@b-1 ~]$ mpirun -np 2 -hostfile
~/hostfile -mca btl self,tcp -mca pml ob1 -mca
btl_tcp_if_include eno1 ./mpitest
And the hang is with the following output:
(eno1 is one of the gigEth interfaces, that
takes OOB traffic as well)
Hello from b-1
The world has 2 nodes
My rank is 0
Hello from b-2
The world has 2 nodes
My rank is 1
Note that if I uncomment the #if 0 - #endif
(i.e. comment out the MPI_Send()/MPI_Recv()
part, the program runs to completion. Also
note that the printfs following
MPI_Send()/MPI_Recv() do not show up on console.
Upon attaching gdb, the stack trace from the
master node is as follows:
Missing separate debuginfos, use:
debuginfo-install glibc-2.17-78.el7.x86_64
libpciaccess-0.13.4-2.el7.x86_64
(gdb) bt
#0 0x00007f72a533eb7d in poll () from
/lib64/libc.so.6
#1 0x00007f72a4cb7146 in poll_dispatch
(base=0xee33d0, tv=0x7fff81057b70)
at poll.c:165
#2 0x00007f72a4caede0 in
opal_libevent2022_event_base_loop (base=0xee33d0,
flags=2) at event.c:1630
#3 0x00007f72a4c4e692 in opal_progress () at
runtime/opal_progress.c:171
#4 0x00007f72a0d07ac1 in opal_condition_wait (
c=0x7f72a5bb1e00 <ompi_request_cond>,
m=0x7f72a5bb1d80 <ompi_request_lock>)
at ../../../../opal/threads/condition.h:76
#5 0x00007f72a0d07ca2 in
ompi_request_wait_completion (req=0x113eb80)
at ../../../../ompi/request/request.h:383
#6 0x00007f72a0d09cd5 in mca_pml_ob1_send
(buf=0x7fff81057db0, count=4,
datatype=0x601080 <ompi_mpi_char>, dst=1, tag=1,
sendmode=MCA_PML_BASE_SEND_STANDARD,
comm=0x601280 <ompi_mpi_comm_world>)
at pml_ob1_isend.c:251
#7 0x00007f72a58d6be3 in PMPI_Send
(buf=0x7fff81057db0, count=4,
type=0x601080 <ompi_mpi_char>, dest=1, tag=1,
comm=0x601280 <ompi_mpi_comm_world>) at psend.c:78
#8 0x0000000000400afa in main (argc=1,
argv=0x7fff81057f18) at mpitest.c:19
(gdb)
And the backtrace on the non-master node is:
(gdb) bt
#0 0x00007ff3b377e48d in nanosleep () from
/lib64/libc.so.6
#1 0x00007ff3b37af014 in usleep () from
/lib64/libc.so.6
#2 0x00007ff3b0c922de in
OPAL_PMIX_PMIX120_PMIx_Fence (procs=0x0, nprocs=0,
info=0x0, ninfo=0) at
src/client/pmix_client_fence.c:100
#3 0x00007ff3b0c5f1a6 in pmix120_fence
(procs=0x0, collect_data=0)
at pmix120_client.c:258
#4 0x00007ff3b3cf8f4b in ompi_mpi_finalize ()
at runtime/ompi_mpi_finalize.c:242
#5 0x00007ff3b3d23295 in PMPI_Finalize () at
pfinalize.c:47
#6 0x0000000000400958 in main (argc=1,
argv=0x7fff785e8788) at mpitest.c:30
(gdb)
The hostfile is as follows:
[durga@b-1 ~]$ cat hostfile
10.4.70.10 slots=1
10.4.70.11 slots=1
#10.4.70.12 slots=1
And the ifconfig output from the master node
is as follows (the other node is similar; all
the IP interfaces are in their respective
subnets) :
[durga@b-1 ~]$ ifconfig
eno1:
flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu
1500
inet 10.4.70.10 netmask 255.255.255.0
broadcast 10.4.70.255
inet6 fe80::21e:c9ff:fefe:13df
prefixlen 64 scopeid 0x20<link>
ether 00:1e:c9:fe:13:df txqueuelen
1000 (Ethernet)
RX packets 48215 bytes 27842846 (26.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52746 bytes 7817568 (7.4 MiB)
TX errors 0 dropped 0 overruns 0
carrier 0 collisions 0
device interrupt 16
eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:1e:c9:fe:13:e0 txqueuelen
1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0
carrier 0 collisions 0
device interrupt 17
lf0:
flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu
2016
inet 192.168.1.2 netmask 255.255.255.0
broadcast 192.168.1.255
inet6 fe80::3002:ff:fe33:3333
prefixlen 64 scopeid 0x20<link>
ether 32:02:00:33:33:33 txqueuelen
1000 (Ethernet)
RX packets 10 bytes 512 (512.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22 bytes 1536 (1.5 KiB)
TX errors 0 dropped 0 overruns 0
carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid
0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 26 bytes 1378 (1.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26 bytes 1378 (1.3 KiB)
TX errors 0 dropped 0 overruns 0
carrier 0 collisions 0
Please help me with this. I am stuck with the
TCP transport, which is the most basic of all
transports.
Thanks in advance
Durga
1% of the executables have 99% of CPU privilege!
Userspace code! Unite!! Occupy the kernel!!!
On Tue, Apr 12, 2016 at 9:32 PM, Gilles
Gouaillardet <gil...@rist.or.jp
<mailto:gil...@rist.or.jp>> wrote:
This is quite unlikely, and fwiw, your
test program works for me.
i suggest you check your 3 TCP networks
are usable, for example
$ mpirun -np 2 -hostfile ~/hostfile -mca
btl self,tcp -mca pml ob1 --mca
btl_tcp_if_include xxx ./mpitest
in which xxx is a [list of] interface name :
eth0
eth1
ib0
eth0,eth1
eth0,ib0
...
eth0,eth1,ib0
and see where problem start occuring.
btw, are your 3 interfaces in 3 different
subnet ? is routing required between two
interfaces of the same type ?
Cheers,
Gilles
On 4/13/2016 7:15 AM, dpchoudh . wrote:
Hi all
I have reported this issue before, but
then had brushed it off as something that
was caused by my modifications to the
source tree. It looks like that is not
the case.
Just now, I did the following:
1. Cloned a fresh copy from master.
2. Configured with the following flags,
built and installed it in my two-node
"cluster".
--enable-debug --enable-debug-symbols
--disable-dlopen
3. Compiled the following program,
mpitest.c with these flags: -g3 -Wall -Wextra
4. Ran it like this:
[durga@smallMPI ~]$ mpirun -np 2
-hostfile ~/hostfile -mca btl self,tcp
-mca pml ob1 ./mpitest
With this, the code hangs at
MPI_Barrier() on both nodes, after
generating the following output:
Hello world from processor smallMPI, rank
0 out of 2 processors
Hello world from processor bigMPI, rank 1
out of 2 processors
smallMPI sent haha!
bigMPI received haha!
<Hangs until killed by ^C>
Attaching to the hung process at one node
gives the following backtrace:
(gdb) bt
#0 0x00007f55b0f41c3d in poll () from
/lib64/libc.so.6
#1 0x00007f55b03ccde6 in poll_dispatch
(base=0x70e7b0, tv=0x7ffd1bb551c0) at
poll.c:165
#2 0x00007f55b03c4a90 in
opal_libevent2022_event_base_loop
(base=0x70e7b0, flags=2) at event.c:1630
#3 0x00007f55b02f0144 in opal_progress ()
at runtime/opal_progress.c:171
#4 0x00007f55b14b4d8b in
opal_condition_wait (c=0x7f55b19fec40
<ompi_request_cond>, m=0x7f55b19febc0
<ompi_request_lock>) at
../opal/threads/condition.h:76
#5 0x00007f55b14b531b in
ompi_request_default_wait_all (count=2,
requests=0x7ffd1bb55370,
statuses=0x7ffd1bb55340) at
request/req_wait.c:287
#6 0x00007f55b157a225 in
ompi_coll_base_sendrecv_zero (dest=1,
stag=-16, source=1, rtag=-16,
comm=0x601280 <ompi_mpi_comm_world>)
at base/coll_base_barrier.c:63
#7 0x00007f55b157a92a in
ompi_coll_base_barrier_intra_two_procs
(comm=0x601280 <ompi_mpi_comm_world>,
module=0x7c2630) at
base/coll_base_barrier.c:308
#8 0x00007f55b15aafec in
ompi_coll_tuned_barrier_intra_dec_fixed
(comm=0x601280 <ompi_mpi_comm_world>,
module=0x7c2630) at
coll_tuned_decision_fixed.c:196
#9 0x00007f55b14d36fd in PMPI_Barrier
(comm=0x601280 <ompi_mpi_comm_world>) at
pbarrier.c:63
#10 0x0000000000400b0b in main (argc=1,
argv=0x7ffd1bb55658) at mpitest.c:26
(gdb)
Thinking that this might be a bug in
tuned collectives, since that is what the
stack shows, I ran the program like this
(basically adding the ^tuned part)
[durga@smallMPI ~]$ mpirun -np 2
-hostfile ~/hostfile -mca btl self,tcp
-mca pml ob1 -mca coll ^tuned ./mpitest
It still hangs, but now with a different
stack trace:
(gdb) bt
#0 0x00007f910d38ac3d in poll () from
/lib64/libc.so.6
#1 0x00007f910c815de6 in poll_dispatch
(base=0x1a317b0, tv=0x7fff43ee3610) at
poll.c:165
#2 0x00007f910c80da90 in
opal_libevent2022_event_base_loop
(base=0x1a317b0, flags=2) at event.c:1630
#3 0x00007f910c739144 in opal_progress ()
at runtime/opal_progress.c:171
#4 0x00007f910db130f7 in
opal_condition_wait (c=0x7f910de47c40
<ompi_request_cond>, m=0x7f910de47bc0
<ompi_request_lock>)
at
../../../../opal/threads/condition.h:76
#5 0x00007f910db132d8 in
ompi_request_wait_completion
(req=0x1b07680) at
../../../../ompi/request/request.h:383
#6 0x00007f910db1533b in mca_pml_ob1_send
(buf=0x0, count=0,
datatype=0x7f910de1e340 <ompi_mpi_byte>,
dst=1, tag=-16,
sendmode=MCA_PML_BASE_SEND_STANDARD,
comm=0x601280 <ompi_mpi_comm_world>) at
pml_ob1_isend.c:259
#7 0x00007f910d9c3b38 in
ompi_coll_base_barrier_intra_basic_linear
(comm=0x601280 <ompi_mpi_comm_world>,
module=0x1b092c0) at
base/coll_base_barrier.c:368
#8 0x00007f910d91c6fd in PMPI_Barrier
(comm=0x601280 <ompi_mpi_comm_world>) at
pbarrier.c:63
#9 0x0000000000400b0b in main (argc=1,
argv=0x7fff43ee3a58) at mpitest.c:26
(gdb)
The mpitest.c program is as follows:
#include <mpi.h>
#include <stdio.h>
#include <string.h>
int main(int argc, char** argv)
{
int world_size, world_rank, name_len;
char
hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s,
rank %d out of %d processors\n",
hostname, world_rank, world_size);
if (world_rank == 1)
{
MPI_Recv(buf, 6, MPI_CHAR, 0, 99,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);
printf("%s received %s\n", hostname,
buf);
}
else
{
strcpy(buf, "haha!");
MPI_Send(buf, 6, MPI_CHAR, 1, 99,
MPI_COMM_WORLD);
printf("%s sent %s\n", hostname, buf);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
The hostfile is as follows:
10.10.10.10 slots=1
10.10.10.11 slots=1
The two nodes are connected by three
physical and 3 logical networks:
Physical: Gigabit Ethernet, 10G iWARP,
20G Infiniband
Logical: IP (all 3), PSM (Qlogic
Infiniband), Verbs (iWARP and Infiniband)
Please note again that this is a fresh,
brand new clone.
Is this a bug (perhaps a side effect of
--disable-dlopen) or something I am doing
wrong?
Thanks
Durga
We learn from history that we never learn
from history.
_______________________________________________
users mailing list
us...@open-mpi.org
<mailto:us...@open-mpi.org>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2016/04/28930.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
