Instead of posting a single, big MPI_Igather, why not post and simultaneously progress multiple, small MPI_Igathers? In this way, you can pipeline multiple outstanding collectives and do post collective processing as soon as requests complete. Without network hardware offload capability, Gilles' observation about the perceived overlap not being truly asynchronous is true; however, you will be able to at least leverage more scalable logarithmic algorithms (depending on which library implementation you use), and make more effective use of CPU cycles.
Josh On Wed, Oct 21, 2015 at 8:31 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote: > Eric, > > #1 MPI_Igather uses only one MPI_Request, and it is marked as completed > when all data has been received. > so no, you cannot process data as it is received. > (and btw, that would require extra subroutines to ask which piece of data > has already been received) > > #2 maybe not ... > a tree based approach has O(log(n)) scaling > (compared to O(n) scaling with your linear method. > so at scale, MPI_Igather will hopefully scale better (and especially if > you are transmitting small messages) > > #3 difficult question ... > first, keep in mind there is currently no progress thread in Open MPI. > that means messages can be received only when MPI_Wait* or MPI_Test* is > invoked. you might hope messages are received when doing some computation > (overlap of computation and communication) but unfortunatly, that does not > happen most of the time. > > linear gather does not scale well (see my previous comment) plus you > openmpi might malloc some space "under the hood" so MPI_Igather will > hopefully scale better. > is there any hard reason why you are using non blocking collective ? > if your application is known to be highly asynchronous and some message > might arrive (way) later than others, and computation is quite expensive, > then your approach might be a good fit. > if your application is pretty synchronous, and cost of computation that > might overlap with communication is not significant, your approach might > have little benefits and poor scalability, so MPI_Gather (not MPI_Igather > since you might have no computation that could overlap with communication) > might be a better choice. > > Cheers, > > Gilles > > > On 10/22/2015 4:46 AM, Eric Chamberland wrote: > > Hi, > > A long time ago (in 2002) we programmed here a non-blocking MPI_Igather > with equivalent calls to MPI_Isend/MPI_Irecv (see the 2 attached files). > > A very convenient advantage of this version, is that I can do some work on > the root process as soon as it start receiving data... Then, it wait for > the next communication to terminate and process the received data. > > Now, I am looking at MPI_Igather (and all non-blocking collective MPI > calls), and I am somewhat surprised (or ignorant) that I cannot have the > root rank receive some data, then process it, then wait until the next > "MPI_irecv" terminate... > > In other words, a MPI_Igather generate only 1 MPI_Request but I would like > to have either "p" (with p = size of communicator) MPI_Request generated or > be able to call "p" times MPI_WaitAny with the same MPI_Request... Am I > normal? :) > > So my 3 questions are: > > #1- Is there a way to use MPI_Igather with MPI_WaitAny (or something > else?) to process data as it is received? > > #2- Big question: will our implementation with MPI_Isend/MPI_Irecv scale > to a large number of processes? What are the possible drawbacks of doing > it like we did? > > #3- Why should I replace our implementation by the native MPI_Igather? > > Thanks! > > Eric > > > > _______________________________________________ > users mailing listus...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/10/27914.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/10/27915.php >
