Il 26/06/2013 02:31, Michael R. Hines ha scritto: > On 06/25/2013 05:06 PM, Paolo Bonzini wrote: >> Il 25/06/2013 22:56, Michael R. Hines ha scritto: >>> I was wrong - this does require a protocol extension. >>> >>> This is because the RDMA transfers are asynchronous, and thus >>> we cannot know in advance that it is safe to unregister the memory >>> associated with each individual transfer before the transfer actually >>> completes. >>> >>> While the destination currently uses the protocol to participate in >>> *registering* the page, the destination does not participate in the >>> RDMA transfers themselves, only the source does, and thus would >>> require a new exchange of messages to block and instruct the >>> destination to unpin the memory. >> Yes, that's what I recalled too (really what mst told me :)). Does it >> need to be blocking though? As long as the pinning is blocking, and >> messages are processed in order, the source can proceed immediately >> after sending an unpin message. This assumes of course that the chunk >> is not being transmitted, and I am not sure how easy the source can >> determine that. > > No, they're not processed in order. In fact, not only does the device > write out of order, but also the PCI bus writes out of order. > This was such a problem in fact, that I fixed several bugs as a result > a few weeks ago (v7 of the patch with an in-depth description). > > The destination simply cannot assume whatsoever what the ordering > of the writes are - that's really the whole point of using RDMA in the > first place so that the software can get out of the way of the transfer > process to lower the latency of each transfer.
The memory is processed out of order, but what about the messages? Those must be in order. Note that I wrote above "This assumes of course that the chunk is not being transmitted". Can the source know when an asynchronous transfer finished, and delay the unpinning until that time? Paolo > > The only option is to send a blocking message to the other side to > request the unpinning (in addition to unpinning on the source first upon > completion of the original transfer). > > As you can expect, this would be very expensive and we must ensure > that we have *very* good a-priori information that this memory will > not need to be re-registered anytime in the near future. > > - Michael > > >
