Re: [OMPI users] IB bandwidth vs. kernels

2007-01-19 Thread Robin Humble
On Thu, Jan 18, 2007 at 03:10:15PM +0200, Gleb Natapov wrote: >On Thu, Jan 18, 2007 at 07:17:13AM -0500, Robin Humble wrote: >> On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: >> >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >> >> On Wed, Jan 17, 2007 at 08:55:31AM -0

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Scott Atchley
On Jan 18, 2007, at 8:11 AM, Peter Kjellstrom wrote: with Lustre, which is about 55% of the theoretical 20 Gb/s advertised speed. I think this should be calculated against 16 Gbps, not 20 Gbps. What is the advertised speed of a IB DDR card? http://mellanox.com/products/hca_cards.php http://

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Peter Kjellstrom
On Thursday 18 January 2007 13:08, Scott Atchley wrote: ... > The best uni-directional performance I have heard of for PCIe 8x IB > DDR is ~1,400 MB/s (11.2 Gb/s) This is on par with what I have seen. > with Lustre, which is about 55% of the > theoretical 20 Gb/s advertised speed. I think this

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Gleb Natapov
On Thu, Jan 18, 2007 at 07:17:13AM -0500, Robin Humble wrote: > On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: > >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: > >> On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: > >> >On Jan 17, 2007, at 2:39 AM, Gl

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
On Thu, Jan 18, 2007 at 11:08:04AM +0200, Gleb Natapov wrote: >On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >> On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >> >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >> >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robi

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Scott Atchley
On Jan 18, 2007, at 5:05 AM, Peter Kjellstrom wrote: On Thursday 18 January 2007 09:52, Robin Humble wrote: ... is ~10Gbit the best I can expect from 4x DDR IB with MPI? some docs @HP suggest up to 16Gbit (data rate) should be possible, and I've heard that 13 or 14 has been achieved before.

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Peter Kjellstrom
On Thursday 18 January 2007 09:52, Robin Humble wrote: ... > is ~10Gbit the best I can expect from 4x DDR IB with MPI? > some docs @HP suggest up to 16Gbit (data rate) should be possible, and > I've heard that 13 or 14 has been achieved before. but those might be > verbs numbers, or maybe horsepowe

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Gleb Natapov
On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: > On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: > >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: > >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: > >>> basically I'm seeing wildly different bandwi

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
argh. attached. cheers, robin On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote: >On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >>On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >>> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: basically I'm s

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-18 Thread Robin Humble
On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote: >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: >>> basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR >>> when I use different kernels. >> Try t

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-17 Thread Brian W. Barrett
On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote: Hi Robin, On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: so this isn't really an OpenMPI questions (I don't think), but you guys will have hit the problem if anyone has... basically I'm seeing wildly different bandwidths over

Re: [OMPI users] IB bandwidth vs. kernels

2007-01-17 Thread Gleb Natapov
Hi Robin, On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote: > > so this isn't really an OpenMPI questions (I don't think), but you guys > will have hit the problem if anyone has... > > basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR > when I use different kern