On Sat, Jan 16, 2016 at 12:03 PM, Thor Lancelot Simon <t...@panix.com> wrote: > On Fri, Jan 15, 2016 at 09:55:37PM +0000, Taylor R Campbell wrote: >> >> If ether_input is going to distribute packets to other CPUs at some >> point anyway, it seems to me that it's a waste of time to do that >> twice -- aside from the issue of IPI scarcity. So perhaps we should >> have either >> >> (a) a per-CPU pktq that never distributes packets to another CPU, or >> (b) a single-CPU pktq, to be used only from the CPU to which the >> device's (queue's) interrupt handler is bound. >> >> I suppose (a) is just a softint together with a percpu(9) of ifqs, and >> (b) is just a softint together with an ifq. > > One consideration here is that with MSI and hardware packet steering > on NICs (for example, according to a hash of the header data up to > and including L4), the ideal situation is that you never, or almost never, > should be distributing packets to a different CPU. The packet should > come in, the appropriate CPU should take the interrupt, and protocol > processing should run to completion on that CPU -- ideally, even ensuring > the receiving process is next scheduled on that same CPU. > > Solaris manages to do this much of the time, and in some configurations > Windows does too (this is why many adapters support the "Toeplitz" hash > Windows uses for their equivalent of our PCB lookup keys). > > But this is ideal. When this isn't possible, I echo Taylor's comment > that you don't want to move processing of a given packet between CPUs > more than once, and I think it belongs as early in the receive path as > possible.
Our most important target is IP forwarding so that I think we can utilize HW multi-queue feature and interrupt distribution; we don't need to distribution packets between CPUs by software (at least at this point). I think we would need software-based packet distribution when we run NetBSD as a server, i.e., we need to deliver packets to a different CPU where a server process which is for the packets is running. ozaki-r
