[OMPI users] OMPI collectives

George I found the info I think you were referring to. Thanks. I then experimented essentially randomly with different algorithms for all reduce. But the issue with really bad performance for certain message sizes persisted with v1.1. The good news is that the upgrade to 1.2 fixed my worst problem

Re: [OMPI users] dma using infiniband protocol

Ha, yeah, I should have been more clear there. I'm simply writing an MPI application. Thanks, Brian On 11/2/06, Jeff Squyres wrote: It depends on what you're trying to do. Are you writing new components internal to Open MPI, or are you just trying to leverage OMPI's PML for some other proj

Re: [OMPI users] tickets 39 & 55

On Nov 2, 2006, at 3:18 PM, Michael Kluskens wrote: So "large" was an attempt to provide *some* of the interfaces -- but [your] experience has shown that this can do more harm than good (i.e., make some legal MPI applications uncompilable because we provide *some* interfaces to MPI_GATHER, but

Re: [OMPI users] dma using infiniband protocol

It depends on what you're trying to do. Are you writing new components internal to Open MPI, or are you just trying to leverage OMPI's PML for some other project? Or are you writing MPI applications? Or ...? On Nov 2, 2006, at 2:22 PM, Brian Budge wrote: Thanks for the pointer, it was

Re: [OMPI users] tickets 39 & 55

All this seems a terrific effort. Is it really justified, especially if it can't cope with the diversity of real-world applications ? I suspect that people who are clever enough to code complex parallel codes involving collective primitives might be able to check arguments. If possible, I su

Re: [OMPI users] dma using infiniband protocol

On Thu, Nov 02, 2006 at 11:57:16AM -0800, Brian Budge wrote: > Thanks for the help guys. > > In my case the memory will be allocated and pinned by my other device > driver. Is it safe to simply use that memory? My pages won't be unpinned > as a result? > If your driver plays nicely with openib

Re: [OMPI users] tickets 39 & 55

On Nov 2, 2006, at 11:53 AM, Jeff Squyres wrote: Adding Craig Rasmussen from LANL into the CC list... On Oct 31, 2006, at 10:26 AM, Michael Kluskens wrote: OpenMPI tickets 39 & 55 deal with problems with the Fortran 90 large interface with regards to: #39: MPI_IN_PLACE in MPI_REDUCE

Re: [OMPI users] dma using infiniband protocol

Thanks for the help guys. In my case the memory will be allocated and pinned by my other device driver. Is it safe to simply use that memory? My pages won't be unpinned as a result? As far as registration, I am sure that OpenMPI will do a better job of that than I could, so I won't even attemp

Re: [OMPI users] dma using infiniband protocol

On Thu, Nov 02, 2006 at 10:37:24AM -0800, Brian Budge wrote: > Hi all - > > I'm wondering how DMA is handled in OpenMPI when using the infiniband > protocol. In particular, will I get a speed gain if my read/write buffers > are already pinned via mlock? > No you will not. mlock has nothing to do

Re: [OMPI users] dma using infiniband protocol

Locking a page with mlock() is not all that is required for RDMA using InfiniBand (or Myrinet, for that matter). You have to call that device's registration function first. In Open MPI, that can be done implicitly with the mpi_leave_pinned option, which will pin memory as needed and then

Re: [OMPI users] dma using infiniband protocol

Thanks for the pointer, it was a very interesting read. It seems that by default OpenMPI uses the nifty pipelining trick with pinning pages while transfer is happening. Also the pinning can be (somewhat) perminant and the state is cached so that next usage requires no registration. I guess it i

Re: [OMPI users] dma using infiniband protocol

This paper explains it pretty well: http://www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols/ On Nov 2, 2006, at 1:37 PM, Brian Budge wrote: Hi all - I'm wondering how DMA is handled in OpenMPI when using the infiniband protocol. In particular, will I get a speed gain if my re

[OMPI users] dma using infiniband protocol

Hi all - I'm wondering how DMA is handled in OpenMPI when using the infiniband protocol. In particular, will I get a speed gain if my read/write buffers are already pinned via mlock? Thanks, Brian

Re: [OMPI users] tickets 39 & 55

Adding Craig Rasmussen from LANL into the CC list... On Oct 31, 2006, at 10:26 AM, Michael Kluskens wrote: OpenMPI tickets 39 & 55 deal with problems with the Fortran 90 large interface with regards to: #39: MPI_IN_PLACE in MPI_REDUCE #55: MPI

Re: [OMPI users] OMPI Collectives

Tony, What do mean by TCP ?  Are you using an ethernet interconnect ? I have noticed a similar slowdown using LAM/MPI and  MPI_Alltoall primitive on our Solaris 10 cluster using gigabit ethernet and TCP. For a large number of nodes I could ever come to a complete hangup. Part of the problem

Re: [OMPI users] MPI_Comm_spawn multiple bproc support

I truly appreciate your patience. Let me talk to some of our Bproc folks and see if they can tell me what is going on. I agree - I would have expected the NODES to be 0,1. The fact that you are getting just 0 explains the behavior you are seeing with Open MPI. I also know (though I don't the comma

[OMPI users] Re: Re: Re: Re: Re:MPI_Comm_spawn multiple bproc support

I again Ralf, >I gather you have access to bjs? Could you use bjs to get a node allocation, >and then send me a printout of the environment? I have slightly changed my cluster configuration for something like: master is running on a machine call: machine10 node 0 is running on a machine call: ma

[OMPI users] Scalapack testing fails with openmpi

Hi, I have a SEGV problem with Scalapack. The same configuration works fine with MPICH, but I seem to get much better performance with Openmpi on this machine. I have attached the log and slmake.inc I am using. I have a the same problem with programs that call this routine that xcdblu uses. It

[OMPI users] openmpi problem

sir, in four machine of our college i have installed in this way..that i m sending u i start four machine from root... then i installed the openmpi1.1.1 -tar.gz using the commands. tar -xvzf openmpi-1.1.1 cd openmpi-1.1.1 ./configure --prefix=/usr/local make make all install ompi_

Re: [OMPI users] MPI_Comm_spawn multiple bproc support

I gather you have access to bjs? Could you use bjs to get a node allocation, and then send me a printout of the environment? All I need to see is what your environment looks like - how does the system tell you what nodes you have been allocated? Then we can make something that will solve your prob

Re: [OMPI users] MPI_Comm_spawn multiple bproc support

Thank you for your support Ralf, I really appreciate. I have now a better understanding of your very first answer asking if I had a NODES environment variable. It was related to the fact that your platform is configured with LSF. I have read some tutorials about LSF and it seems that LSF provide