On 12.11.2012 22:47, Karim wrote:
On 12-11-12 03:27 PM, Andre Oppermann wrote:
On 12.11.2012 20:15, Karim wrote:
Hi all,
I have been following the current discussions on igb tx/rx locking with great
interest and I though
I would point out (as it was before pointed out in kern/138392) that any driver
setting if_start to
NULL and using if_transmit (multi-queue or not) will effectively break ALTQ.
The reason for this can be found in tbr_timeout() in altq_subr.c where the
token bucket regulator
goes through all ALTQ registered interface and tries to 'kick' them into
sending through if_start().
tbr_timeout() :
...
for (ifp = TAILQ_FIRST(&V_ifnet); ifp;
ifp = TAILQ_NEXT(ifp, if_list)) {
/* read from if_snd unlocked */
if (!TBR_IS_ENABLED(&ifp->if_snd))
continue;
active++;
if (!IFQ_IS_EMPTY(&ifp->if_snd) &&
ifp->if_start != NULL) /* if_start is NULL
if if_transmit is
used in em/igb driver */
(*ifp->if_start)(ifp);
}
...
As you can see if_start is NULL on those new multi-queue enabled drivers which
has for net effect to
'break' ALTQ's token bucket regulator.
I am writing this because I am interested in your comments on how this can be
fixed properly looking
forward. The whole range of suggestions; from 'don't compile with EM_MULTIQUEUE
defined' to 'here is
how you can make ALTQ use drbr' will help.
This whole area needs some serious re-consideration for 10.0.
Even without the if_start issue ALTQ is somewhat broken because
the DMA rings nowadays are just too deep.
See this recent message to this list for some thoughts:
http://lists.freebsd.org/pipermail/freebsd-net/2012-November/033780.html
On top of refining the stack/driver boundary I'm thinking of a
dedicated ethernet layer to consolidate all the ethernet extensions,
have a single place to go and to prevent the drivers from re-inventing
the wheel all over every time.
I'm working towards it and leaning into various drivers while trying
to bring in the hybrid interrupt/polling mode with life-lock prevention.
It'll take a few weeks until I'm ready to show a stack/ethernet/driver
prototype for discussion. Parts may surface earlier in my tcp_workqueue
svn branch.
Hi,
Glad to see someone looking into this :)
The addition of a common layer for queuing algorithm is an interesting idea and
makes me wonder how
alternate queuing techniques would be able to use this shim layer to leverage
devices with multiple
queues.
The current limitation (in terms of performances) with ALTQ going forward is
its current inability
to use multiple queues, in part because it was left out when the current (drbr)
implementation of
multiqueues was made but mainly because of its current internal structure
(using a single global IFQ
lock).
Is it part of the plan to abstract multiqueuing from alternate queuing
algorithm or will it still be
left to ALTQ, through driver managed queues for example, to manage the TX DMA
ring lock?
I don't know yet how exactly it will look like. ALTQ, in modified
form, will remain part of the functionality set. Most multi-queue
network cards also support various modes of queue arbitration for
different service classes. This may be leveraged for ALTQ as well.
There's many different variants of multi-queue usage depending on
the goal of the overall system. I try to allow them to co-exist and
to be selectable at run-time while remaining at a sane complexity level.
--
Andre
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
