There is no async progress in Open MPI at this time so this is the expected behavior. We plan to fix this for the 1.9 release series.
-Nathan Hjelm
HPC-5, LANL
On Mon, Apr 07, 2014 at 11:12:06AM +0800, Zehan Cui wrote:
> Hi Matthieu,
>
> Thanks for your suggestion. I tried MPI_Waitall(), but the results are
> the same. It seems the communication didn't overlap with computation.
>
> Regards,
> Zehan
>
> On 4/5/14, Matthieu Brucher <matthieu.bruc...@gmail.com> wrote:
> > Hi,
> >
> > Try waiting on all gathers at the same time, not one by one (this is
> > what non blocking collectives are made for!)
> >
> > Cheers,
> >
> > Matthieu
> >
> > 2014-04-05 10:35 GMT+01:00 Zehan Cui <zehan....@gmail.com>:
> >> Hi,
> >>
> >> I'm testing the non-blocking collective of OpenMPI-1.8.
> >>
> >> I have two nodes with Infiniband to perform allgather on totally 128MB
> >> data.
> >>
> >> I split the 128MB data into eight pieces, and perform computation and
> >> MPI_Iallgatherv() on one piece of data each iteration, hoping that the
> >> MPI_Iallgatherv() of last iteration can be overlapped with computation of
> >> current iteration. A MPI_Wait() is called at the end of last iteration.
> >>
> >> However, the total communication time (including the final wait time) is
> >> similar with that of the traditional blocking MPI_Allgatherv, even
> >> slightly
> >> higher.
> >>
> >>
> >> Following is the test pseudo-code, the source code are attached.
> >>
> >> ===========================
> >>
> >> Using MPI_Allgatherv:
> >>
> >> for( i=0; i<8; i++ )
> >> {
> >> // computation
> >> mytime( t_begin );
> >> computation;
> >> mytime( t_end );
> >> comp_time += (t_end - t_begin);
> >>
> >> // communication
> >> t_begin = t_end;
> >> MPI_Allgatherv();
> >> mytime( t_end );
> >> comm_time += (t_end - t_begin);
> >> }
> >> --------------------------------------------
> >>
> >> Using MPI_Iallgatherv:
> >>
> >> for( i=0; i<8; i++ )
> >> {
> >> // computation
> >> mytime( t_begin );
> >> computation;
> >> mytime( t_end );
> >> comp_time += (t_end - t_begin);
> >>
> >> // communication
> >> t_begin = t_end;
> >> MPI_Iallgatherv();
> >> mytime( t_end );
> >> comm_time += (t_end - t_begin);
> >> }
> >>
> >> // wait for non-blocking allgather to complete
> >> mytime( t_begin );
> >> for( i=0; i<8; i++ )
> >> MPI_Wait;
> >> mytime( t_end );
> >> wait_time = t_end - t_begin;
> >>
> >> ==============================
> >>
> >> The results of Allgatherv is:
> >> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
> >> --host
> >> gnode102,gnode103 ./Allgatherv 128 2 | grep time
> >> Computation time : 8481279 us
> >> Communication time: 319803 us
> >>
> >> The results of Iallgatherv is:
> >> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
> >> --host
> >> gnode102,gnode103 ./Iallgatherv 128 2 | grep time
> >> Computation time : 8479177 us
> >> Communication time: 199046 us
> >> Wait time: 139841 us
> >>
> >>
> >> So, does this mean that current OpenMPI implementation of MPI_Iallgatherv
> >> doesn't support offloading of collective communication to dedicated cores
> >> or
> >> network interface?
> >>
> >> Best regards,
> >> Zehan
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> users mailing list
> >> us...@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> >
> >
> > --
> > Information System Engineer, Ph.D.
> > Blog: http://matt.eifelle.com
> > LinkedIn: http://www.linkedin.com/in/matthieubrucher
> > Music band: http://liliejay.com/
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
>
> --
> Best Regards
> Zehan Cui(崔泽汉)
> -----------------------------------------------------------
> Institute of Computing Technology, Chinese Academy of Sciences.
> No.6 Kexueyuan South Road Zhongguancun,Haidian District Beijing,China
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
pgpccAMD3gv69.pgp
Description: PGP signature
