Hi,
Try waiting on all gathers at the same time, not one by one (this is
what non blocking collectives are made for!)
Cheers,
Matthieu
2014-04-05 10:35 GMT+01:00 Zehan Cui <zehan....@gmail.com>:
> Hi,
>
> I'm testing the non-blocking collective of OpenMPI-1.8.
>
> I have two nodes with Infiniband to perform allgather on totally 128MB data.
>
> I split the 128MB data into eight pieces, and perform computation and
> MPI_Iallgatherv() on one piece of data each iteration, hoping that the
> MPI_Iallgatherv() of last iteration can be overlapped with computation of
> current iteration. A MPI_Wait() is called at the end of last iteration.
>
> However, the total communication time (including the final wait time) is
> similar with that of the traditional blocking MPI_Allgatherv, even slightly
> higher.
>
>
> Following is the test pseudo-code, the source code are attached.
>
> ===========================
>
> Using MPI_Allgatherv:
>
> for( i=0; i<8; i++ )
> {
> // computation
> mytime( t_begin );
> computation;
> mytime( t_end );
> comp_time += (t_end - t_begin);
>
> // communication
> t_begin = t_end;
> MPI_Allgatherv();
> mytime( t_end );
> comm_time += (t_end - t_begin);
> }
> --------------------------------------------
>
> Using MPI_Iallgatherv:
>
> for( i=0; i<8; i++ )
> {
> // computation
> mytime( t_begin );
> computation;
> mytime( t_end );
> comp_time += (t_end - t_begin);
>
> // communication
> t_begin = t_end;
> MPI_Iallgatherv();
> mytime( t_end );
> comm_time += (t_end - t_begin);
> }
>
> // wait for non-blocking allgather to complete
> mytime( t_begin );
> for( i=0; i<8; i++ )
> MPI_Wait;
> mytime( t_end );
> wait_time = t_end - t_begin;
>
> ==============================
>
> The results of Allgatherv is:
> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
> gnode102,gnode103 ./Allgatherv 128 2 | grep time
> Computation time : 8481279 us
> Communication time: 319803 us
>
> The results of Iallgatherv is:
> [cmy@gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
> gnode102,gnode103 ./Iallgatherv 128 2 | grep time
> Computation time : 8479177 us
> Communication time: 199046 us
> Wait time: 139841 us
>
>
> So, does this mean that current OpenMPI implementation of MPI_Iallgatherv
> doesn't support offloading of collective communication to dedicated cores or
> network interface?
>
> Best regards,
> Zehan
>
>
>
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/
