On Tue, Jan 30, 2001 at 10:19:58AM -0700, Jeff V. Merkey wrote:
> On Mon, Jan 29, 2001 at 09:41:21PM -0700, Todd wrote:
Todd,
I just got back some more numbers from Dolphin. The newer D330
LC3 chipsets are running at 667 MB/S Link speed, and on a
Serverworks HE system, we are seeing 240 MB/S throughput via
the newer PCI-SCI adapters. I have some D330 adapters on
the way here, and will repost newer numbers after the Alpha
changes are rolled in and I repost the drivers again next
week sometime.
On 32 bit PCI, the average we are seeing going userpace -> userspace is
120-140 MB/S ranges in those systems that have a PCI bus with
bridge chipsets that can support these data rates.
That's 2 x G-Enet.
:-)
Jeff
>
> Todd,
>
> I ran the tests on a box that has a limit of 70MB/S PCI throughput.
> There are GaAs (Gallium Arsenide) implementations of SCI that run
> well into the Gigabyte Ranges, and I've seen some of this hardware.
> The NUMA-Q chipsets in Sequents cluster boxes are actually SCI.
> Sun Microsystems uses SCI as their clustering interconnect for their
> Sparc servers. The adapters these drivers I posted support are a bi-CMOS
> implementation of the SCI LC3 chipsets, and even though they are
> bi-CMOS, the Link speed on the back end is still 500 MB/S --
> very respectable.
>
> The PCI-SCI cards these drivers support in PCI systems have been clocked up
> to 140 MB/S+ on those systems that have enough bandwidth to actually
> push this much data. These boards can pump up to 500MB/S over the
> SCI fabric, however, current PCI technology doesn't allow you to
> push this much data. I also tested on the D320 chipsets, the newer
> D330 chipsets on the PSB66 cards support the 66Mhz bus and have been
> measured up to the Max PCI speeds.
>
> The PCI-SCI adapters run circles around G-Enet on systems that can
> really pump this much data through the PCI bus. Also, the numbers I
> posted are doing push/pull DMA transfers between user_space -> user_space
> in another system with **NO COPYING**. Ethernet and LAN networking always
> copies data into userspace -- SCI has the ability to dump it directly
> into user space pages without copying. That's what is cool about SCI,
> you can pump this data around with almost no processor utilization --
> important on a cluster if you are doing computational stuff -- you need
> every cycle you can squeeze, and don't want to waste them copying
> data all over the place. Sure, G-Enet can pump 124 MB/S, but the
> processor utilitzation will be high, and there will be lots of
> copying going on in the system. What numbers does G-Enet provide
> doing userspace -> userspace transfers, and at what processor
> overhead? These are the types of things that are the metrics for
> a good comparison. Also, G-Enet has bandwidth limitations, the
> SCI standard does not, it's only limited by the laws of Physics
> (which are being reached in the Dolphin Labs in Norway).
>
> The GaAs SCI technology I have seen has hop latencies in the SCI
> switches @ 16 nano-seconds to route a packet, with xfer rates into
> the Gigabytes per second -- very fast and low latency.
>
> These cards will use whatever PCI bandwidth is present in the host
> system, up to 500 MB/S. As the PCI bus gets better, nice to know
> SCI is something that will keep it's value, since 500 MB/S gives us
> a lot of room to grow into.
>
> I could ask Dolphin for a GaAs version of the LC3 card (one board would
> cost the equivalent to the income of a small third world nation), and
> rerun the tests on a Sparc system or Sequent system, and watch G-Enet
> system suck wind in comparison.
>
> :-)
>
> I posted the **ACCURATE** numbers from my test, but I did clarify that I
> was using a system with a limp PCI bus.
>
> Jeff
>
>
> > folx,
> >
> > i must be missing something here. i'm not aware of a PCI bus that only
> > supports 70 MBps but i am probably ignorant. this is why i was confused
> > by jeff's performance numbers. 33MHz 32-bit PCI busses should do around
> > 120MB/s (just do the math 33*32/8 allowing for some overhead of PCI bus
> > negotiation), much greater than the numbers jeff is reporting. 66 MHz
> > 64bit busses should do on the order of 500MB/s.
> >
> > the performance numbers that jeff is reporting are not very impressive
> > even for the slowest PCI bus. we're seeing 993 Mbps (124MB/s) using the
> > alteon acenic gig-e cards on 32-bit cards on a 66MHz bus. i would expect
> > to get somewhat slower on a 33MHz bus but not catastrophically so
> > (certainly nothing as slow as 60MB/s or 480Mb/s).
> >
> > what am i misunderstanding here?
> >
> > todd
> >
> > On Mon, 29 Jan 2001, Jeff V. Merkey wrote:
> >
> > > Date: Mon, 29 Jan 2001 16:49:53 -0700
> > > From: Jeff V. Merkey <[EMAIL PROTECTED]>
> > > To: [EMAIL PROTECTED]
> > > Cc: [EMAIL PROTECTED]
> > > Subject: Re: [ANNOUNCE] Dolphin PCI-SCI RPM Drivers 1.1-4 released
> > >
> > >
> > > Relative to some performance questions folks have asked, the SCI
> > > adapters are limited by PCI bus speeds. If your system supports
> > > 64-bit PCI you get much higher numbers. If you have a system
> > > that supports 100+ Megabyte/second PCI throughput, the SCI
> > > adapters will exploit it.
> > >
> > > This test was performed in on a 32-bit PCI system with a PCI bus
> > > architecture that's limited to 70 MB/S.
> > >
> > > Jeff
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to [EMAIL PROTECTED]
> > > Please read the FAQ at http://www.tux.org/lkml/
> > >
> >
> > =========================================================
> > Todd Underwood, [EMAIL PROTECTED]
> >
> > criticaltv.com
> > news, analysis and criticism. about tv.
> > and other stuff.
> >
> > =========================================================
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [EMAIL PROTECTED]
> Please read the FAQ at http://www.tux.org/lkml/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
