My reproducer is below (SCATTERV only). It needs to be compiled with 64-bit
default reals, and I'm running on four cores of a single linux86-64 box running
SLED 12.3 (except where noted).
Using Open-MPI with different compilers:
With g95: The non-root procs print the correct values, but the root process seg
faults somewhere inside the SCATTERV call.
With portland: I get: -1614907703: __hpf_esend: not implemented
(All procs print out the correct values.)
With Intel (on a Mac Pro): Complains about a null communicator in MPI_FINALIZE
and crashes. All procs print out the correct values.
With all three of these compilers, if I comment out the entire IF (MYPN.EQ.0)
code so that all procs pass RARR1 into both the send and recv buffers, I get no
errors.
With gfortran: This works either way (with IN_PLACE or without).
Other MPI implementations:
With MPICH2 (any compiler) and Intel Visual Fortran on Windows, the IN_PLACE
code works.
They specifically prohibit passing RARR1 into both the send and recv buffers on
the root proc.
Reproducer:
PROGRAM MAIN
IMPLICIT NONE
REAL, DIMENSION(1200) :: RARR1
INTEGER, DIMENSION(4) :: SEND_NUM, SEND_OFF
INTEGER :: RECV_NUM, MYPN, NPES, IERR
INTEGER :: I, J
INCLUDE 'mpif.h'
SEND_NUM = (/ 300, 300, 300, 300 /)
SEND_OFF = (/ 0, 300, 600, 900 /)
RECV_NUM = 300
CALL MPI_INIT(IERR)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD, NPES, IERR)
CALL MPI_COMM_RANK(MPI_COMM_WORLD, MYPN, IERR)
IF (MYPN.EQ.0) THEN
DO I = 1,1200
RARR1(I) = 0.001*I
ENDDO
ELSE
RARR1 = 0.0
ENDIF
IF (MYPN.EQ.0) THEN
CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
MPI_IN_PLACE,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
ELSE
CALL MPI_SCATTERV(RARR1,SEND_NUM,SEND_OFF,MPI_DOUBLE_PRECISION, &
RARR1,RECV_NUM,MPI_DOUBLE_PRECISION,0,MPI_COMM_WORLD,IERR)
ENDIF
OPEN(71+MYPN,FORM='FORMATTED',POSITION='APPEND')
WRITE(71+MYPN,'(3E15.7)') RARR1(1:300)
CLOSE(71+MYPN)
CALL MPI_FINALIZE(IERR)
END PROGRAM MAIN
________________________________________
From: users [users-boun...@open-mpi.org] on behalf of Nathan Hjelm
[hje...@lanl.gov]
Sent: Wednesday, October 09, 2013 12:37 PM
To: Open MPI Users
Subject: Re: [OMPI users] MPI_IN_PLACE with GATHERV, AGATHERV, and SCATERV
These functions are tested nightly and there has been no indication any of these
functions fail with MPI_IN_PLACE. Can you provide a reproducer?
-Nathan
HPC-3, LANL
On Tue, Oct 08, 2013 at 07:40:50PM +0000, Gerlach, Charles A. wrote:
> I have an MPI code that was developed using MPICH1 and OpenMPI before the
> MPI2 standards became commonplace (before MPI_IN_PLACE was an option).
>
>
>
> So, my code has many examples of GATHERV, AGATHERV and SCATTERV, where I
> pass the same array in as the SEND_BUF and the RECV_BUF, and this has
> worked fine for many years.
>
>
>
> Intel MPI and MPICH2 explicitly disallow this behavior according to the
> MPI2 standard. So, I have gone through and used MPI_IN_PLACE for all the
> GATHERV/SCATTERVs that used to pass the same array twice. This code now
> works with MPICH2 and Intel_MPI, but fails with OpenMPI-1.6.5 on multiple
> platforms and compilers.
>
>
>
> PLATFORM COMPILER SUCCESS? (For at least one
> simple example)
>
> ------------------------------------------------------------
>
> SLED 12.3 (x86-64) - Portland group - fails
>
> SLED 12.3 (x86-64) - g95 - fails
>
> SLED 12.3 (x86-64) - gfortran - works
>
>
>
> OS X 10.8 -- intel -fails
>
>
>
>
>
> In every case where OpenMPI fails with the MPI_IN_PLACE code, I can go
> back to the original code that passes the same array twice instead of
> using MPI_IN_PLACE, and it is fine.
>
>
>
> I have made a test case doing an individual GATHERV with MPI_IN_PLACE, and
> it works with OpenMPI. So it looks like there is some interaction with my
> code that is causing the problem. I have no idea how to go about trying to
> debug it.
>
>
>
>
>
> In summary:
>
>
>
> OpenMPI-1.6.5 crashes my code when I use GATHERV, AGATHERV, and SCATTERV
> with MPI_IN_PLACE.
>
> Intel MPI and MPICH2 work with my code when I use GATHERV, AGATHERV, and
> SCATTERV with MPI_IN_PLACE.
>
>
>
> OpenMPI-1.6.5 works with my code when I pass the same array to SEND_BUF
> and RECV_BUF instead of using MPI_IN_PLACE for those same GATHERV,
> AGATHERV, and SCATTERVs.
>
>
>
>
>
> -Charles
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
