strange igb interface performance problems

[Mario Spinthiras]

Hello All,

 The subject topic might catch a few by surprise because for the most part
it smells a lot like a problem that came to light in 2009 regarding bad
performance on Intel NICs using the igb driver.

 I currently have a point to point link between 2 pfsense 2.0RC1 running
FreeBSD 8.1 that connect via a 1Gbps link. The link itself has been tested
and the provider reckons that the link is fine. (I also believe this too).

We then started high and went low to search for the problem. We ditched out
the media converters, went on a mad mission to tune every single last detail
on the stack to give us the full performance, however on the 1Gbps line we
see a rough 20Mbps.

We then turned towards the igb driver which smells a little funny
considering that the issues I mentioned that have emerged back in 2009
regarding similar issues are alive and well today.

Does anyone out there have any experience or knowledge of such problems
caused by the igb driver? I'm going to plan a maintenance window on this
link within the next day or so hoping that I'll be able to live boot a 2.0
RC1 nightly and try it again. The current RC1 is one of the earlier ones.

Any info you can send my way on this baffling mystery that has taken me on a
goose chase. Disabling TSO/LRO does not solve the problem for me. Can anyone
shed some light on this?


Thanking you all in advance,

Mario
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: strange igb interface performance problems

[Mario Spinthiras]

tal)
10241/5957/16198/25600 mbuf clusters in use (current/cache/total/max)
10240/5120 mbuf+clusters out of packet secondary zone in use (current/cache)
0/3989/3989/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
25612K/33173K/58785K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
[2.0-RC1][root@pfSense.localdomain]/root(30):

Node B:

[2.0-RC1][root@pfsense.localdomain]/root(50): netstat -m
9691/4274/13965 mbufs in use (current/cache/total)
6355/1891/8246/25600 mbuf clusters in use (current/cache/total/max)
6354/1070 mbuf+clusters out of packet secondary zone in use (current/cache)
1216/1271/2487/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
20091K/9934K/30026K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/8/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
[2.0-RC1][root@pfsense.localdomain]/root(51):


dmesg output on both nodes.

Node A:

igb1:  port
0xe880-0xe89f mem
0xf97e-0xf97f,0xf9c0-0xf9ff,0xf97dc000-0xf97d irq 17 at
device 0.1 on pci3
igb1: Using MSIX interrupts with 5 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]


Node B:

igb3:  port
0xc880-0xc89f mem
0xf9fe-0xf9ff,0xfa00-0xfa3f,0xf9fdc000-0xf9fd irq 37 at
device 0.1 on pci6
igb3: Using MSIX interrupts with 5 vectors
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: [ITHREAD]
igb3: link state changed to UP
igb3_vlan593: link state changed to UP


pciconf output:

Node A:

igb1@pci0:3:0:1:class=0x02 card=0xa03c8086 chip=0x10c98086
rev=0x01hdr=0x00
class  = network
subclass   = ethernet
bar   [10] = type Memory, range 32, base 0xf97e, size 131072,
enabled
bar   [14] = type Memory, range 32, base 0xf9c0, size 4194304,
enabled
bar   [18] = type I/O Port, range 32, base 0xe880, size 32, enabled
bar   [1c] = type Memory, range 32, base 0xf97dc000, size 16384, enabled


Node B:

igb3@pci0:6:0:1:class=0x02 card=0xa02b8086 chip=0x10e88086
rev=0x01 hdr=0x00
class  = network
subclass   = ethernet
bar   [10] = type Memory, range 32, base 0xf9fe, size 131072,
enabled
bar   [14] = type Memory, range 32, base 0xfa00, size 4194304,
enabled
bar   [18] = type I/O Port, range 32, base 0xc880, size 32, enabled
bar   [1c] = type Memory, range 32, base 0xf9fdc000, size 16384, enabled


I've been looking at issues regarding performance with the igb issue from a
few years back at this thread :
http://lists.freebsd.org/pipermail/freebsd-doc/2009-June/015983.html. These
are very similar problems to what I'm having however disabling LRO and TSO
does not work for me.

I'm quite frustrated with this because I'm not getting an error or not
looking in the right place. Can someone out there point me in the right
direction? Any help will be much appreciated.

Warm Regards,
Mario Spinthiras
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: strange igb interface performance problems

[Mario Spinthiras]

Hi,

 To disable LRO I tried ifconfig igb1 -lro and I also tried adding the
sysctl params in loader.conf. Both did not result in any change.

 I'm going to have to find a way to use the new igb driver I guess, hoping
that would fix it maybe.


Regards,
Mario
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"