Re: [OMPI users] IB bandwidth vs. kernels

2007-01-19 Thread Robin Humble
On Thu, Jan 18, 2007 at 03:10:15PM +0200, Gleb Natapov wrote: >On Thu, Jan 18, 2007 at 07:17:13AM -0500, Robin Humble wrote: >> On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: >> >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >> >> On Wed, Jan 17, 2007 at 08:55:31AM -0

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Scott Atchley
On Jan 18, 2007, at 8:11 AM, Peter Kjellstrom wrote: with Lustre, which is about 55% of the theoretical 20 Gb/s advertised speed. I think this should be calculated against 16 Gbps, not 20 Gbps. What is the advertised speed of a IB DDR card? http://mellanox.com/products/hca_cards.php http://

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Peter Kjellstrom
On Thursday 18 January 2007 13:08, Scott Atchley wrote: ... > The best uni-directional performance I have heard of for PCIe 8x IB > DDR is ~1,400 MB/s (11.2 Gb/s) This is on par with what I have seen. > with Lustre, which is about 55% of the > theoretical 20 Gb/s advertised speed. I think this

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Gleb Natapov
On Thu, Jan 18, 2007 at 07:17:13AM -0500, Robin Humble wrote: > On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: > >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: > >> On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: > >> >On Jan 17, 2007, at 2:39 AM, Gl

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >> On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >> >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >> >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robi

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Scott Atchley
On Jan 18, 2007, at 5:05 AM, Peter Kjellstrom wrote: On Thursday 18 January 2007 09:52, Robin Humble wrote: ... is ~10Gbit the best I can expect from 4x DDR IB with MPI? some docs @HP suggest up to 16Gbit (data rate) should be possible, and I've heard that 13 or 14 has been achieved before.

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Peter Kjellstrom
On Thursday 18 January 2007 09:52, Robin Humble wrote: ... > is ~10Gbit the best I can expect from 4x DDR IB with MPI? > some docs @HP suggest up to 16Gbit (data rate) should be possible, and > I've heard that 13 or 14 has been achieved before. but those might be > verbs numbers, or maybe horsepowe

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Gleb Natapov
On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: > On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: > >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: > >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: > >>> basically I'm seeing wildly different bandwi

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
argh. attached. cheers, robin On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >>On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >>> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: basically I'm s

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: >>> basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR >>> when I use different kernels. >> Try t

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-17 Thread Brian W. Barrett
On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: Hi Robin, On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: so this isn't really an OpenMPI questions (I don't think), but you guys will have hit the problem if anyone has... basically I'm seeing wildly different bandwidths over

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-17 Thread Gleb Natapov
Hi Robin, On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: > > so this isn't really an OpenMPI questions (I don't think), but you guys > will have hit the problem if anyone has... > > basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR > when I use different kern

[OMPI users] IB bandwidth vs. kernels

2007-01-17 Thread Robin Humble
so this isn't really an OpenMPI questions (I don't think), but you guys will have hit the problem if anyone has... basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR when I use different kernels. I'm testing with netpipe-3.6.2's NPmpi, but a home-grown pingpong sees the same