Re: [OMPI users] OS X, OpenMPI 1.1: An error occurred in MPI_Allreduce on, communicator MPI_COMM_WORLD (Jeff Squyres (jsquyres))

Frank Kahle Wed, 5 Jul 2006 13:14:28 -0400

users-requ...@open-mpi.org wrote:

A few clarifying questions:


What is your netmask on these hosts?

Where is the MPI_ALLREDUCE in your app -- right away, or somewhere deep
within the application?  Can you replicate this with a simple MPI
application that essentially calls MPI_INIT, MPI_ALLREDUCE, and
MPI_FINALIZE?

Can you replicate this with a simple MPI app that does an MPI_SEND /

MPI_RECV between two processes on the different subnets?

Thanks.


@ Jeff,

netmask 255.255.255.0

Running a simple "hello world" yields no error on each subnet, butrunning "hello world" on both subnets yields the error


[g5dual.3-net:00436] *** An error occurred in MPI_Send
[g5dual.3-net:00436] *** on communicator MPI_COMM_WORLD
[g5dual.3-net:00436] *** MPI_ERR_INTERN: internal error
[g5dual.3-net:00436] *** MPI_ERRORS_ARE_FATAL (goodbye)

Hope this helps!

Frank


Just in case you wanna check the source:
c    Fortran example hello_world
     program hello
     include 'mpif.h'
     integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)
     character*12 message

     call MPI_INIT(ierror)
     call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
     call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
     tag = 100

     if (rank .eq. 0) then
       message = 'Hello, world'
       do i=1, size-1
         call MPI_SEND(message, 12, MPI_CHARACTER, i, tag,
    &                  MPI_COMM_WORLD, ierror)
       enddo

     else
       call MPI_RECV(message, 12, MPI_CHARACTER, 0, tag,
    &                MPI_COMM_WORLD, status, ierror)
     endif

     print*, 'node', rank, ':', message
     call MPI_FINALIZE(ierror)
     end


or the full output:

[powerbook:/Network/CFD/hello] motte% mpirun -d -np 5 --hostfile./hostfile /Network/CFD/hello/hello_world

[powerbook.2-net:00606] [0,0,0] setting up session dir with
[powerbook.2-net:00606]         universe default-universe
[powerbook.2-net:00606]         user motte
[powerbook.2-net:00606]         host powerbook.2-net
[powerbook.2-net:00606]         jobid 0
[powerbook.2-net:00606]         procid 0

[powerbook.2-net:00606] procdir:/tmp/openmpi-sessions-motte@powerbook.2-net_0/default-universe/0/0[powerbook.2-net:00606] jobdir:/tmp/openmpi-sessions-motte@powerbook.2-net_0/default-universe/0[powerbook.2-net:00606] unidir:/tmp/openmpi-sessions-motte@powerbook.2-net_0/default-universe

[powerbook.2-net:00606] top: openmpi-sessions-motte@powerbook.2-net_0
[powerbook.2-net:00606] tmp: /tmp

[powerbook.2-net:00606] [0,0,0] contact_file/tmp/openmpi-sessions-motte@powerbook.2-net_0/default-universe/universe-setup.txt

[powerbook.2-net:00606] [0,0,0] wrote setup file
[powerbook.2-net:00606] pls:rsh: local csh: 1, local bash: 0
[powerbook.2-net:00606] pls:rsh: assuming same remote shell as local shell
[powerbook.2-net:00606] pls:rsh: remote csh: 1, remote bash: 0
[powerbook.2-net:00606] pls:rsh: final template argv:

[powerbook.2-net:00606] pls:rsh: /usr/bin/ssh <template> orted--debug --bootproxy 1 --name <template> --num_procs 6 --vpid_start 0--nodename <template> --universe motte@powerbook.2-net:default-universe--nsreplica "0.0.0;tcp://192.168.2.3:49443" --gprreplica"0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0

[powerbook.2-net:00606] pls:rsh: launching on node Powerbook.2-net