Hi,
I'm trying to track down the cause of poor network performance under
OpenBSD4.0/i386 on HP Proliants (DL380-G4 and DL360-G4p), which seems
to be concerning ethernet 802.3x flow control on the bge NICs.
Test topology is:
HP DL380-G4
int bge0 (BCM5704C auto at 1000baseT full-duplex)
|
|
int Gig 13/6 (auto at 1000baseT full-duplex)
Cisco 6513 chassis + WS-X6548-GE-TX + WS-X6748-GE-TX
int Gig 12/47 (auto at 1000baseT full-duplex)
|
|
int bge0 (BCM5704C auto at 1000baseT full-duplex)
HP DL360-G4p
Test traffic is generated with:
On Source: dd if=/dev/zero bs=1k count=10000 | nc _peer_ 1234
On Sink: nc -l 1234 > /dev/null
With 4.0-release kernel (GENERIC#1107), the bge driver does not
negotiate flowcontrol with the switch:
switch# show interfaces flowcontrol | inc Port|admin|Gi12/47|Gi13/6
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
Gi12/47 desired off desired off 0 0
Gi13/6 desired off desired off 0 0
Network traffic is very slow and the receiving host reports
significant 'Input errors' on the NIC interface after transfer:
source~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:18:fe:32:2e:4a 1050 0
1276 0 0
source~> dd if=/dev/zero bs=1k count=100000 | nc _peer_ 1234
100000+0 records in
100000+0 records out
102400000 bytes transferred in 13.219 secs (7746244 bytes/sec)
source~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:18:fe:32:2e:4a 52684 0
73166 0 0
sink~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:17:a4:45:f5:25 79 0
106 0 0
sink~> nc -l 1234 > /dev/null
sink~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:17:a4:45:f5:25 70841 11
50894 0 0
With 4.0-snapshot kernel (GENERIC#1362), the bge driver now
negotiates flow control:
switch# show interfaces flowcontrol | inc Port|admin|Gi12/47|Gi13/6
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
Gi12/47 desired on desired on 0 0
Gi13/6 desired on desired on 0 0
However, the transfer is still very slow, and the receiving host
still reports significant 'Input errors' on the NIC interface after
transfer:
source~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:18:fe:32:2e:4a 1459 0
1762 0 0
source~> dd if=/dev/zero bs=1k count=100000 | nc _peer_ 1234
100000+0 records in
100000+0 records out
102400000 bytes transferred in 14.120 secs (7251650 bytes/sec)
source ~>netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:18:fe:32:2e:4a 53240 0
73457 0 0
sink~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:17:a4:45:f5:25 89 0
98 0 0
sink~> nc -l 1234 > /dev/null
sink~> netstat -i -I bge0 | grep -e Name -e Link
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls
bge0 1500 <Link> 00:17:a4:45:f5:25 70849 9
51186 0 0
To summarise, it seems as though flow-control is negotiated for both
TX & RX in the recent bge driver, but is only functional for TX (if
at all). The only relevant source change I can find is:
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_bge.c.diff?
r1=1.202&r2=1.203&f=h
"Flow control support for bge(4)/brgphy(4). From brad@ based on code
fromNetBSD"
with includes the comment /* We can do both TXPAUSE and RXPAUSE. */
Setting 'ifconfig bge0 debug' provides no additional output. I have
also repeated the tests with serveral differnet servers, NICs (all
bge) and cables and switches to remove faulty device issues.
Has anyone an ideas on fixes for this, or how to debug the issue
further ?
Dmesg below
/Pete
# dmesg
OpenBSD 4.0-current (GENERIC) #1362: Fri Feb 9 14:26:43 MST 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 3.40GHz ("GenuineIntel" 686-class) 3.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,
CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-
CPL,EST,CNXT-ID,CX16
real mem = 1073258496 (1048104K)
avail mem = 970895360 (948140K)
using 4256 buffers containing 53784576 bytes (52524K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+ BIOS, date 12/31/99, BIOS32 rev. 0 @
0xf0000, SMBIOS rev. 2.3 @ 0xec000 (77 entries)
bios0: HP ProLiant DL380 G4
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
pcibios0: PCI BIOS has 7 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC"
rev 0x00)
pcibios0: PCI bus #10 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4000! 0xcc000/0x1600
0xee000/0x2000!
acpi at mainbus0 not configured
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x0c
ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x0c
pci1 at ppb0 bus 2
ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci2 at ppb1 bus 3
bge0 at pci2 dev 1 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704
B0 (0x2100): irq 5, address 00:17:a4:45:f5:25
brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge1 at pci2 dev 1 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704
B0 (0x2100): irq 5, address 00:17:a4:45:f5:24
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
ppb2 at pci1 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci3 at ppb2 bus 4
ciss0 at pci3 dev 3 function 0 "Compaq Smart Array 64xx" rev 0x01: irq 5
ciss0: 1 LD, HW rev 1, FW 2.58/2.58
scsibus0 at ciss0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.58> SCSI0 0/
direct fixed
sd0: 700049MB, 700049 cyl, 64 head, 32 sec, 512 bytes/sec, 1433700592
sec total
ppb3 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x0c
pci4 at ppb3 bus 5
ppb4 at pci4 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci5 at ppb4 bus 6
ppb5 at pci4 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci6 at ppb5 bus 10
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: irq 5
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: irq 5
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: irq 5
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3 at pci0 dev 29 function 3 "Intel 82801EB/ER USB" rev 0x02: irq 5
usb3 at uhci3: USB revision 1.0
uhub3 at usb3
uhub3: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: irq 5
usb4 at ehci0: USB revision 2.0
uhub4 at usb4
uhub4: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
ppb6 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci7 at ppb6 bus 1
vga1 at pci7 dev 3 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Compaq iLO" rev 0x01 at pci7 dev 4 function 0 not configured
"Compaq iLO" rev 0x01 at pci7 dev 4 function 2 not configured
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
DMA, channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <TSSTcorp, CD-ROM TS-L162C, N204> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask efe5 netmask efe5 ttymask ffe7
pctr: user-level cycle counter enabled
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
#
# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:17:a4:45:f5:25
groups: egress
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet6 fe80::217:a4ff:fe45:f525%bge0 prefixlen 64 scopeid 0x1
inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.x
#
