On Thu, Nov 02, 2006 at 11:57:16AM -0800, Brian Budge wrote: > Thanks for the help guys. > > In my case the memory will be allocated and pinned by my other device > driver. Is it safe to simply use that memory? My pages won't be unpinned > as a result? > If your driver plays nicely with openib driver everything should be OK. If by pinned you mean mlcok() then your are save since openib doesn't use mlock().
> As far as registration, I am sure that OpenMPI will do a better job of that > than I could, so I won't even attempt to futz with that. > > Thanks, > Brian > > On 11/2/06, Brian W Barrett <bbarr...@lanl.gov> wrote: > > > >Locking a page with mlock() is not all that is required for RDMA > >using InfiniBand (or Myrinet, for that matter). You have to call > >that device's registration function first. In Open MPI, that can be > >done implicitly with the mpi_leave_pinned option, which will pin > >memory as needed and then leave it pinned for the life of the > >buffer. Or it can be done ahead of time by calling MPI_ALLOC_MEM. > > > >Because the amount of memory a NIC can have pinned at any time may > >not directly match the total amount of memory that can be mlock()ed > >at any given time, it's also not a safe assumption that a buffer > >allocated with MPI_ALLOC_MEM or used with an RDMA transfer from MPI > >is going to be mlock()ed as a side effect of NIC registration. Open > >MPI internally might unregister that memory with the NIC in order to > >register a different memory segment for another memory transfer. > > > >Brian > > > > > >On Nov 2, 2006, at 12:22 PM, Brian Budge wrote: > > > >> Thanks for the pointer, it was a very interesting read. > >> > >> It seems that by default OpenMPI uses the nifty pipelining trick > >> with pinning pages while transfer is happening. Also the pinning > >> can be (somewhat) perminant and the state is cached so that next > >> usage requires no registration. I guess it is possible to use pre- > >> pinned memory, but do I need to do anything special to do so? I > >> will already have some buffers pinned to allow DMAs to devices > >> across PCI-Express, so it makes sense to use one pinned buffer so > >> that I can avoid memcpys. > >> > >> Are there any HOWTO tutorials or anything? I've searched around, > >> but it's possible I just used the wrong search terms. > >> > >> Thanks, > >> Brian > >> > >> > >> > >> On 11/2/06, Jeff Squyres <jsquy...@cisco.com> wrote: This paper > >> explains it pretty well: > >> > >> http://www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols/ > >> > >> > >> > >> On Nov 2, 2006, at 1:37 PM, Brian Budge wrote: > >> > >> > Hi all - > >> > > >> > I'm wondering how DMA is handled in OpenMPI when using the > >> > infiniband protocol. In particular, will I get a speed gain if my > >> > read/write buffers are already pinned via mlock? > >> > > >> > Thanks, > >> > Brian > >> > _______________________________________________ > >> > users mailing list > >> > us...@open-mpi.org > >> > http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> -- > >> Jeff Squyres > >> Server Virtualization Business Unit > >> Cisco Systems > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > >_______________________________________________ > >users mailing list > >us...@open-mpi.org > >http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Gleb.
