On Sat, Dec 23, 2006 at 09:36:33PM +0000, Robert Watson wrote: > > On Sat, 23 Dec 2006, John Polstra wrote: > > >>That said, dropping and regrabbing the driver lock in the rxeof routine > >>of any driver is bad. It may be safe to do, but it incurs horrible > >>performance penalties. It essentially allows the time-critical, high > >>priority RX path to be constantly preempted by the lower priority > >>if_start or if_ioctl paths. Even without this preemption and priority > >>inversion, you're doing an excessive number of expensive lock ops in the > >>fast path. > > > >We currently make this a lot worse than it needs to be by handing off the > >received packets one at a time, unlocking and relocking for every packet. > >It would be better if the driver's receive interrupt handler would harvest > >all of the incoming packets and queue them locally. Then, at the end, hand > >off the linked list of packets to the network stack wholesale, unlocking > >and relocking only once. (Actually, the list could probably be handed off > >at the very end of the interrupt service routine, after the driver has > >already dropped its lock.) We wouldn't even need a new primitive, if > >ether_input() and the other if_input() functions were enhanced to deal > >with a possible list of packets instead of just a single one. > > I try this experiement every few years, and generally don't measure much > improvement. I'll try it again with 10gbps early next year once back in > the office again. The more interesting transition is between the link > layer and the network layer, which is high on my list of topics to look > into in the next few weeks. In particular, reworking the ifqueue handoff. > The tricky bit is balancing latency, overhead, and concurrency... > > FYI, there are several sets of patches floating around to modify if_em to > hand off queues of packets to the link layer, etc. They probably need > updating, of course, since if_em has changed quite a bit in the last year. > In my implementaiton, I add a new input routine that accepts mbuf packet > queues.
I'm just curious, do you remember average length of mbuf queue in your tests? While experimenting with bge(4) driver (taskqueue, interrupt moderation, converted bge_rxeof() to above scheme), i've found it's quite easy to exhaust available mbuf clusters under load (trying to queue hundreids of received packets). So i had to limit rx queue to rather low length. > > Robert N M Watson > Computer Laboratory > University of Cambridge -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- [EMAIL PROTECTED] === ================================================================ _______________________________________________ cvs-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/cvs-all To unsubscribe, send any mail to "[EMAIL PROTECTED]"
