Re: [OMPI users] dma using infiniband protocol

2006-11-03 Thread Jeff Squyres
You shouldn't have to do anything special; just write normal MPI programs. There are a variety of MPI tutorials available on the web; a particularly good one is available here: http://webct.ncsa.uiuc.edu:8900/public/MPI/ I think someone mentioned that you can use MPI_ALLOC_MEM to get pr

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Brian Budge
Ha, yeah, I should have been more clear there. I'm simply writing an MPI application. Thanks, Brian On 11/2/06, Jeff Squyres wrote: It depends on what you're trying to do. Are you writing new components internal to Open MPI, or are you just trying to leverage OMPI's PML for some other proj

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Jeff Squyres
It depends on what you're trying to do. Are you writing new components internal to Open MPI, or are you just trying to leverage OMPI's PML for some other project? Or are you writing MPI applications? Or ...? On Nov 2, 2006, at 2:22 PM, Brian Budge wrote: Thanks for the pointer, it was

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Gleb Natapov
On Thu, Nov 02, 2006 at 11:57:16AM -0800, Brian Budge wrote: > Thanks for the help guys. > > In my case the memory will be allocated and pinned by my other device > driver. Is it safe to simply use that memory? My pages won't be unpinned > as a result? > If your driver plays nicely with openib

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Brian Budge
Thanks for the help guys. In my case the memory will be allocated and pinned by my other device driver. Is it safe to simply use that memory? My pages won't be unpinned as a result? As far as registration, I am sure that OpenMPI will do a better job of that than I could, so I won't even attemp

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Gleb Natapov
On Thu, Nov 02, 2006 at 10:37:24AM -0800, Brian Budge wrote: > Hi all - > > I'm wondering how DMA is handled in OpenMPI when using the infiniband > protocol. In particular, will I get a speed gain if my read/write buffers > are already pinned via mlock? > No you will not. mlock has nothing to do

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Brian W Barrett
Locking a page with mlock() is not all that is required for RDMA using InfiniBand (or Myrinet, for that matter). You have to call that device's registration function first. In Open MPI, that can be done implicitly with the mpi_leave_pinned option, which will pin memory as needed and then

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Brian Budge
Thanks for the pointer, it was a very interesting read. It seems that by default OpenMPI uses the nifty pipelining trick with pinning pages while transfer is happening. Also the pinning can be (somewhat) perminant and the state is cached so that next usage requires no registration. I guess it i

Re: [OMPI users] dma using infiniband protocol

2006-11-02 Thread Jeff Squyres
This paper explains it pretty well: http://www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols/ On Nov 2, 2006, at 1:37 PM, Brian Budge wrote: Hi all - I'm wondering how DMA is handled in OpenMPI when using the infiniband protocol. In particular, will I get a speed gain if my re