--- On Thu, 8/13/09, Barney Cordoba <barney_cord...@yahoo.com> wrote:
> From: Barney Cordoba <barney_cord...@yahoo.com> > Subject: Re: nfe taskq performance issues > To: "Peter Steele" <pste...@webmail.maxiscale.com>, pyu...@gmail.com > Cc: freebsd-net@freebsd.org > Date: Thursday, August 13, 2009, 7:56 AM > > > --- On Wed, 8/12/09, Pyun YongHyeon <pyu...@gmail.com> > wrote: > > > From: Pyun YongHyeon <pyu...@gmail.com> > > Subject: Re: nfe taskq performance issues > > To: "Peter Steele" <pste...@webmail.maxiscale.com> > > Cc: freebsd-net@freebsd.org > > Date: Wednesday, August 12, 2009, 5:35 PM > > On Thu, Jul 23, 2009 at 08:58:07AM > > -0700, Peter Steele wrote: > > > We've been hitting serious nfe taskq performance > > issues during stress > > > tests and in doing some research on the problem > we > > came across this old > > > email: > > > > > > > > > > > > From: Ivan Voras <ivo...@freebsd.org> > > > Date: April 28, 2009 3:53:14 AM PDT > > > To: freebsd-thre...@freebsd.org > > > Cc: freebsd-net@freebsd.org, > > freebsd-performa...@freebsd.org > > > Subject: Re: FreeBSD 7.1 taskq em performance > > > > > > > > I have been hitting some barrier with > FreeBSD 7.1 > > network performance. > > > I > > > > have written an application which contains > two > > kernel threads that > > > takes > > > > mbufs directly from a network interface and > > forwards to another > > > network > > > > interface. This idea is to simulate > different > > network environment. > > > > > > > > I have been using FreeBSD 6.4 amd64 and > tested > > with an Ixia box > > > > (specialised hardware firing very high > packet > > rate). The PC was a > > > Core2 2.6 > > > > GHz with dual ports Intel PCIE Gigabit > network > > card. It can manage up > > > to 1.2 > > > > million pps. > > > > > > > > I have a higher spec PC with FreeBSD 7.1 > amd64 > > and Quadcore 2.3 GHz > > > and > > > > PCIE Gigabit network card. The performance > can > > only achieve up to 600k > > > pps. > > > > I notice the 'taskq em0' and 'taskq em1' is > solid > > 100% CPU but it is > > > not in > > > > FreeBSD 6.4. > > > > > > > > > > > > In our case we are running FreeBSD 7.0, but we > are > > seeing our boxes > > > experience serious thread starvation issues as > the > > nfe0 cpu percentage > > > climbs steadily while cpu idle time drops at > times to > > 0 percent. This > > > email thread mentioned a patch for the em driver > > here: > > > > > > > > > > > > http://people.yandex-team.ru/~wawa/ > > > <http://people.yandex-team.ru/%7Ewawa/> > > > > > > > > > > > > > > Does anyone know if this patch will work with the > nfe > > driver? > > > > > > > That's for em(4). > > > > AFAIK all nfe(4) controllers lacks intelligent > interrupts > > moderation so driver should be prepared to handle > > excessive > > interrupt loads. I'm not sure whether NVIDIA ethernet > > controllers > > really lacks efficient interrupt mitigation mechanism > but > > it > > seems Linux also faces the same hardware problem. > > As you might know there is no publicly available data > sheet > > for > > NVIDIA controllers so setting it right looks very hard > to > > me. > > > > Try removing the INTR_MPSAFE flag from the bus_setup_intr() > call. > The entire point of using filters is to reduce lock > contention. > It might not solve the problem but its clearly an > unnecessary > potential bottleneck. > > Barney I'm curious as to the statistics on your system. Your quad core adapter may actually be hurting the performance. What is the cpu usage distribution shown by top -SH when you are loaded, and how many interrupts/second are you getting per nic? It looks like the default moderation is set to 8000 ints/sec, which is probably ok for 1 interrupt per NIC. Its not clear whether multiple msix interrupts are allocated. Spreading interrupts isn't always a good thing as it increases lock contention so much as to be counterproductive, unless you have properly written mutex management code; which nfe doesn't. Barney _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
