When it comes to intrinsic Fortran-90 functions, or to libraries provided by the compiler vendor [e.g. MKL in the case of Intel], I do agree that they *should* be able to parse the array-section notation and use the correct memory layout.
However, for libraries that are not part of Fortran-90, such as MPI, Lapack, FFTW, etc, etc, which are designed, programmed, and built independently of Fortran-90, I don't see how the F90 compiler can enforce them to comply with the array-section notation and memory layout. These libraries may or may not offer a fully Fortran-90 compliant API. I think MPI doesn't. MPI provides an alternative mechanism through user-defined types, which is not specific to Fortran-90 programs either. FFTW has yet another mechanism, not as broad, and focused on arrays/vectors. Of course the compiler may provide a workaround with copy in/out to/from memory-contiguous temporary arrays, as the Intel compiler does. I wouldn't call it a compiler bug when the compiler doesn't provide this workaround. However, of course the metaphysical nature of what is and is not a bug is certainly debatable. :) Anyway, we seem to agree that this is not an MPI problem. MPI provides its solution. Some compilers provide an alternative solution. One can chose whatever solution that is convenient to solve the problem at hand. Gus Correa On Dec 14, 2011, at 1:04 PM, David Warren wrote: > Actually, sub array passing is part of the F90 standard (at least according > to every document I can find), and not an Intel extension. So if it doesn't > work you should complain to the compiler company. One of the reasons for > using it is that the compiler should be optimized for whatever method they > chose to use. As there are multiple options in the F90 standard for how > arrays get passed, it is not really a good idea to circumvent the official > method. Using user defined data types is great as long as the compiler > chooses to do a simple pointer pass, however if they use the copy in/out > option you will be making much larger temporary arrays than if you just pass > the correct subarray. Anyway, this is not really an MPI issue as much as an > F90 bug in your compiler. > > On 12/14/11 08:57, Gustavo Correa wrote: >> Hi Patrick >> >> > From my mere MPI and Fortran-90 user point of view, >> I think that the solution offered by the MPI standard [at least up to MPI-2] >> to address the problem of non-contiguous memory layouts is to use MPI >> user-defined types, >> as I pointed out in my previous email. >> I like this solution because it is portable and doesn't require the >> allocation of >> temporary arrays, and the additional programming effort is not that big. >> >> As far as I know, MPI doesn't parse or comply with the Fortran-90 >> array-section notation and syntax. All buffers in the MPI calls are >> pointers/addresses to the >> first element on the buffer, which will be tracked according to the number >> of elements passed >> to the MPI call, and according to the MPI type passed to the MPI routine >> [which should be >> a user-defined type, if you need to implement a fancy memory layout]. >> >> That MPI doesn't understand Fortran-90 array-sections doesn't surprise me so >> much. >> I think Lapack doesn't do it either, and many other legitimate Fortran >> libraries don't >> 'understand' array-sections either. >> FFTW, for instance, goes a long way do define its own mechanism to >> specify fancy memory layouts independently of the Fortran-90 array-section >> notation. >> Amongst the libraries with Fortran interfaces that I've used, MPI probably >> provides the most >> flexible and complete mechanism to describe memory layout, through >> user-defined types. >> In your case I think the work required to declare a MPI_TYPE_VECTOR to >> handle your >> table 'tab' is not really big or complicated. >> >> As two other list subscribers mentioned, and you already tried, >> the Intel compiler seems to offer an extension >> to deal with this, and shortcut the use of MPI user-defined types. >> This Intel compiler extension apparently uses under the hood the same idea >> of a >> temporary array that you used programatically in one of the 'bide' program >> versions >> that you sent in your original message. >> The temporary array is used to ship data to/from contiguous/non-contiguous >> memory before/after the MPI call is invoked. >> I presume this Intel compiler extension would work with libraries other than >> MPI, >> whenever the library doesn't understand the Fortran-90 array-section >> notation. >> I never used this extension, though. >> For one thing, this solution may not be portable to other compilers. >> Another aspect to consider is how much 'under the hood memory allocation' >> this solution >> would require if the array you pass to MPI_BCAST is really big, >> and how much this may impact performance. >> >> I hope this helps, >> Gus Correa >> >> On Dec 14, 2011, at 11:03 AM, Patrick Begou wrote: >> >> >>> Thanks all for your anwers. yes, I understand well that it is a non >>> contiguous memory access problem as the MPI_BCAST should wait for a pointer >>> on a valid memory zone. But I'm surprised that with the MPI module usage >>> Fortran does not hide this discontinuity in a contiguous temporary copy of >>> the array. I've spent some time to build openMPI with g++/gcc/ifort (to >>> create the right mpi module) and ran some additional tests: >>> >>> >>> Default OpenMPI is openmpi-1.2.8-17.4.x86_64 >>> >>> # module load openmpi >>> # mpif90 ess.F90&& mpirun -np 4 ./a.out >>> 0 1 2 3 0 1 >>> 2 3 0 1 2 3 >>> 0 1 2 3 >>> # module unload openmpi >>> The result is Ok but sometime it hangs (when I require are a lot of >>> processes) >>> >>> With OpenMPI 1.4.4 and gfortran from gcc-fortran-4.5-19.1.x86_64 >>> >>> # module load openmpi-1.4.4-gcc-gfortran >>> # mpif90 ess.F90&& mpirun -np 4 ./a.out >>> 0 -1 -1 -1 0 -1 >>> -1 -1 0 -1 -1 -1 >>> 0 -1 -1 -1 >>> # module unload openmpi-1.4.4-gcc-gfortran >>> Node 0 only update the global array with it's subarray. (i only print node >>> 0 result) >>> >>> >>> With OpenMPI 1.4.4 and ifort 10.1.018 (yes, it's quite old, I have the >>> latest one but it isn't installed!) >>> >>> # module load openmpi-1.4.4-gcc-intel >>> # mpif90 ess.F90&& mpirun -np 4 ./a.out >>> ess.F90(15): (col. 5) remark: LOOP WAS VECTORIZED. >>> 0 -1 -1 -1 0 -1 >>> -1 -1 0 -1 -1 -1 >>> 0 -1 -1 -1 >>> >>> # mpif90 -check arg_temp_created ess.F90&& mpirun -np 4 ./a.out >>> gives a lot of messages like: >>> forrtl: warning (402): fort: (1): In call to MPI_BCAST1DI4, an array >>> temporary was created for argument #1 >>> >>> So a temporary array is created for each call. So where is the problem ? >>> >>> About the fortran compiler, I'm using similar behavior (non contiguous >>> subarrays) in MPI_sendrecv calls and all is working fine: I ran some >>> intensive tests from 1 to 128 processes on my quad-core workstation. This >>> Fortran solution was easier than creating user defined data types. >>> >>> Can you reproduce this behavior with the test case ? What are your OpenMPI >>> and Gfortran/ifort versions ? >>> >>> Thanks again >>> >>> Patrick >>> >>> The test code: >>> >>> PROGRAM bide >>> USE mpi >>> IMPLICIT NONE >>> INTEGER :: nbcpus >>> INTEGER :: my_rank >>> INTEGER :: ierr,i,buf >>> INTEGER, ALLOCATABLE:: tab(:,:) >>> >>> CALL MPI_INIT(ierr) >>> CALL MPI_COMM_RANK(MPI_COMM_WORLD,my_rank,ierr) >>> CALL MPI_COMM_SIZE(MPI_COMM_WORLD,nbcpus,ierr) >>> >>> ALLOCATE (tab(0:nbcpus-1,4)) >>> >>> tab(:,:)=-1 >>> tab(my_rank,:)=my_rank >>> DO i=0,nbcpus-1 >>> CALL MPI_BCAST(tab(i,:),4,MPI_INTEGER,i,MPI_COMM_WORLD,ierr) >>> ENDDO >>> IF (my_rank .EQ. 0) print*,tab >>> CALL MPI_FINALIZE(ierr) >>> >>> END PROGRAM bide >>> >>> -- =============================================================== | Equipe >>> M.O.S.T. | http://most.hmg.inpg.fr | | Patrick BEGOU | ------------ | | >>> LEGI | mailto:patrick.be...@hmg.inpg.fr | | BP 53 X | Tel 04 76 82 51 35 | >>> | 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 | >>> =============================================================== >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
