I'd check the firewall settings. The stack trace indicates that the one host
is trying to connect to the other (Open MPI initiates non-blocking TCP
connections that can be polled on later).
On Nov 10, 2010, at 12:46 PM, David Zhang wrote:
> Have you double checked your firewall settings, TCP/IP settings, and SSH keys
> are all setup correctly for all machines including the host?
>
> On Wed, Nov 10, 2010 at 2:57 AM, Grzegorz Maj <ma...@wp.pl> wrote:
> Hi all,
> I've got a problem with sending messages from one of my machines. It
> appears during MPI_Send/MPI_Recv and MPI_Bcast. The simplest case I've
> found is two processes, rank 0 sending a simple message and rank 1
> receiving this message. I execute these processes using mpirun with
> -np 2.
> - when both processes are executed on the host machine, it works fine;
> - when both processes are executed on client machines (both on the
> same or different machines), it works fine;
> - when sender is executed on one of the client machines and receiver
> on the host machine, it works fine;
> - when sender is executed on the host machine and receiver on client
> machine, it blocks.
>
> This last case is my problem. When adding option '--mca
> btl_base_verbose 30' to mpirun, I get:
>
> ----------
> [host:28186] mca: base: components_open: Looking for btl components
> [host:28186] mca: base: components_open: opening btl components
> [host:28186] mca: base: components_open: found loaded component self
> [host:28186] mca: base: components_open: component self has no register
> function
> [host:28186] mca: base: components_open: component self open function
> successful
> [host:28186] mca: base: components_open: found loaded component sm
> [host:28186] mca: base: components_open: component sm has no register function
> [host:28186] mca: base: components_open: component sm open function successful
> [host:28186] mca: base: components_open: found loaded component tcp
> [host:28186] mca: base: components_open: component tcp has no register
> function
> [host:28186] mca: base: components_open: component tcp open function
> successful
> [host:28186] select: initializing btl component self
> [host:28186] select: init of component self returned success
> [host:28186] select: initializing btl component sm
> [host:28186] select: init of component sm returned success
> [host:28186] select: initializing btl component tcp
> [host:28186] select: init of component tcp returned success
> [client01:19803] mca: base: components_open: Looking for btl components
> [client01:19803] mca: base: components_open: opening btl components
> [client01:19803] mca: base: components_open: found loaded component self
> [client01:19803] mca: base: components_open: component self has no
> register function
> [client01:19803] mca: base: components_open: component self open
> function successful
> [client01:19803] mca: base: components_open: found loaded component sm
> [client01:19803] mca: base: components_open: component sm has no
> register function
> [client01:19803] mca: base: components_open: component sm open
> function successful
> [client01:19803] mca: base: components_open: found loaded component tcp
> [client01:19803] mca: base: components_open: component tcp has no
> register function
> [client01:19803] mca: base: components_open: component tcp open
> function successful
> [client01:19803] select: initializing btl component self
> [client01:19803] select: init of component self returned success
> [client01:19803] select: initializing btl component sm
> [client01:19803] select: init of component sm returned success
> [client01:19803] select: initializing btl component tcp
> [client01:19803] select: init of component tcp returned success
> 00 of 2 host
> [host:28186] btl: tcp: attempting to connect() to address 10.0.7.97 on
> port 53255
> 01 of 2 client01
> ----------
>
> Where lines "00 of 2 host" and "01 of 2 client01" are just my debug
> saying "mpirank of comm_size hostname". The last but one line appears
> in call to Send:
> MPI::COMM_WORLD.Send(message, 5, MPI::CHAR, 1, 13);
>
> When executing the sender on host with strace, I get:
>
> ----------
> ...
> connect(10, {sa_family=AF_INET, sin_port=htons(1024),
> sin_addr=inet_addr("10.0.7.97")}, 16) = -1 EINPROGRESS (Operation now
> in progress)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}, {fd=10, events=POLLOUT}], 7, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}, {fd=10, events=POLLOUT}], 7, 0) = 1 ([{fd=10,
> revents=POLLOUT}])
> getsockopt(10, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
> send(10, "D\227\0\1\0\0\0\0", 8, 0) = 8
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}, {fd=10, events=POLLIN}], 7, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}, {fd=10, events=POLLIN}], 7, 0) = 1 ([{fd=10,
> revents=POLLIN}])
> recv(10, "", 8, 0) = 0
> close(10) = 0
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}], 6, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}], 6, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}], 6, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}], 6, 0) = 0 (Timeout)
> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
> events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9,
> events=POLLIN}], 6, 0) = 0 (Timeout)
> ...
> (forever)
> ...
> ----------
>
> For me it looks like the above connect is responsible for establishing
> connection, but I'm afraid I don't understand what those calls for
> poll are supposed to do.
>
> Attaching gdb to the sender gives me:
>
> ----------
> (gdb) bt
> #0 0xffffe410 in __kernel_vsyscall ()
> #1 0x0064993b in poll () from /lib/libc.so.6
> #2 0xf7df07b5 in poll_dispatch () from
> /home/gmaj/openmpi/lib/libopen-pal.so.0
> #3 0xf7def8c3 in opal_event_base_loop () from
> /home/gmaj/openmpi/lib/libopen-pal.so.0
> #4 0xf7defbe7 in opal_event_loop () from
> /home/gmaj/openmpi/lib/libopen-pal.so.0
> #5 0xf7de323b in opal_progress () from
> /home/gmaj/openmpi/lib/libopen-pal.so.0
> #6 0xf7c51455 in mca_pml_ob1_send () from
> /home/gmaj/openmpi/lib/openmpi/mca_pml_ob1.so
> #7 0xf7ed9c60 in PMPI_Send () from /home/gmaj/openmpi/lib/libmpi.so.0
> #8 0x0804e900 in main ()
> ----------
>
> If anybody knows what may cause this problem or what may I do to find
> the reason, any help is appreciated.
>
> My open-mpi is version 1.4.1.
>
>
> Regards,
> Grzegorz Maj
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> David Zhang
> University of California, San Diego
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
