I'll defer to the Mellanox guys to reply more in detail, but here's a few thoughts:
- Is MVAPICH using XRC? (I never played with XRC much; it would surprise me if it caused instability on the order of up to 100 micros -- I ask just to see if it is an apples-to-apples comparison) - The nRepeats value in this code is only 10, meaning that it only seems to be doing 10 iterations on each size. For small sizes, this might well be not enough to be accurate. Have you tried increasing it? Or using a different benchmark app, such as NetPIPE, osu_latency, ...etc.? On Feb 16, 2010, at 8:49 AM, Repsher, Stephen J wrote: > Well the "good" news is I can end your debate over binding here...setting > mpi_paffinity_alone 1 did nothing. (And personally as a user, I don't care > what the default is so long as info is readily apparent in the main > docs...and I did see the FAQs on it). > > It did lead me to try another parameter though, -mca mpi_preconnect_all 1, > which seems to reduce the measured latency reliably of subounce, but it's > still sporadic and order ~10-100 microseconds. It leads me to think that > OpenMPI has issues with the method of measurement, which is simply to send > progressively larger blocked messages right after calling MPI_Init (starting > at 0 bytes which it times as the latency). OpenMPI's lazy connections clearly > mess with this. > > But still not consistently 1-2 microsecs... > > Steve > > > -----Original Message----- > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On > Behalf Of Ralph Castain > Sent: Monday, February 15, 2010 11:21 PM > To: Open MPI Users > Subject: Re: [OMPI users] Bad Infiniband latency with subounce > > > On Feb 15, 2010, at 8:44 PM, Terry Frankcombe wrote: > > > On Mon, 2010-02-15 at 20:18 -0700, Ralph Castain wrote: > >> Did you run it with -mca mpi_paffinity_alone 1? Given this is 1.4.1, you > >> can set the bindings to -bind-to-socket or -bind-to-core. Either will give > >> you improved performance. > >> > >> IIRC, MVAPICH defaults to -bind-to-socket. OMPI defaults to no binding. > > > > > > Is this sensible? Won't most users want processes bound? OMPI's > > supposed to "to the right thing" out of the box, right? > > Well, that depends on how you look at it. Been the subject of a lot of debate > within the devel community. If you bind by default and it is a shared node > cluster, then you can really mess people up. On the other hand, if you don't > bind by default, then people that run benchmarks without looking at the > options can get bad numbers. Unfortunately, there is no automated way to tell > if the cluster is configured for shared use or dedicated nodes. > > I honestly don't know that "most users want processes bound". One > installation I was at set binding by default using the system mca param file, > and got yelled at by a group of users that had threaded apps - and most > definitely did -not- want their processes bound. After a while, it became > clear that nothing we could do would make everyone happy :-/ > > I doubt there is a right/wrong answer - at least, we sure can't find one. So > we don't bind by default so we "do no harm", and put out FAQs, man pages, > mpirun option help messages, etc. that explain the situation and tell you > when/how to bind. > > > > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
