On 9 Feb, 2011, at 07:29 , rihad wrote: > Hi, we're a medium sized ISP that need to pass all incoming user traffic > through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until > yesterday it had two 1 gb em cards, one for input, one for output. As we were > approaching the bandwidth limitation we switched the cards for a two-port > Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the > built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only > about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow. > Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat > -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8). > Late in the night yesterday I didn't have enough opportunity to test the > newer FreeBSD under load, but from the time we did and I know, the same > slowness started happening at about 300-400 mbps load. There are no more > fragmentation failed errors. No evident drops as per "netstat -s | fgrep > drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0 > and ix1 are configured at their default settings. > > Then I read something about the number of ixgbe device descriptors > (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to > 4096 permittable. But after some grepping on the source tree I saw that > contrary to what the old docs say they are both set to an optimal value: > > /sys/dev/ixgbe/ixgbe.c: > /* > ** Number of TX descriptors per ring, > ** setting higher than RX as this seems > ** the better performing choice. > */ > static int ixgbe_txd = PERFORM_TXD; > TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd); > > /* Number of RX descriptors per ring */ > static int ixgbe_rxd = PERFORM_RXD; > TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd) > > > /sys/dev/ixgbe/ixgbe.h: > /* > * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the > * number of transmit descriptors allocated by the driver. Increasing this > * value allows the driver to queue more transmits. Each descriptor is 16 > * bytes. Performance tests have show the 2K value to be optimal for top > * performance. > */ > #define DEFAULT_TXD 1024 > #define PERFORM_TXD 2048 > #define MAX_TXD 4096 > #define MIN_TXD 64 > > > > So, here's my kernel config for your viewing pleasure: > include GENERIC > > ident SHAPER > > nomakeoptions DEBUG > > nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4 > nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5 > nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > nooptions COMPAT_FREEBSD32 # Compatible with i386 binaries > > nooptions INET6 # IPv6 communications protocols > options ZERO_COPY_SOCKETS > # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work > #options DEVICE_POLLING > options IPFIREWALL #firewall > options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default > > > Here's /etc/sysctl.conf: > > net.inet.ip.fw.verbose=0 > > kern.ipc.shmall=65536 > kern.ipc.shmmax=268435456 > kern.ipc.semmap=1024 > kern.ipc.nmbclusters=111111 > > net.inet.ip.fastforwarding=1 > net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3?? > net.isr.direct=0 > net.inet.ip.intr_queue_maxlen=5000 > > hw.intr_storm_threshold=9000 > #dev.em.0.rx_processing_limit=-1 # device not used any more > > > > > Any tips? I'll be happy to try and add some more info upon request. > > > Thanks. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
I don't know if it's the same issue, but I had severe performance issues with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on 7.2 too. Regards, Nikolay_______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"