* Jason Healy <jhe...@logn.net> [2009-11-04 16:37]: > The systems work great, but are chewing up about 60% of their time on > interrupts (~9000 according to vmstat, with ~7500 going to the LAN/WAN > cards). This is fine; everything is working and I know that high > interrupt load was inevitable at the time. However, I need to ramp up > the traffic on this system soon (we're at 30Mbps / 3.5kpps right now), > so I want to make sure I can keep the load under control.
you probably don't need to worry. especially with em load doesn't remotely increase linearaly with traffic. > I know that the first thing I should do is upgrade to 4.6, which I > plan to do. However, I'm looking for other "best practices", which > I've divided into two major sections below: yes, 4.6 is MUCH faster than 4.3. > > Interrupt Mitigation: > ===================== > > Since the system is under moderately heavy interrupt load, I'd like to > try and improve that if possible since it seems that's going to be the > first limit I hit on this system. In the "Tuning OpenBSD" paper: > > http://www.openbsd.org/papers/tuning-openbsd.ps > > they mention "sharing interrupts" on a high load system. If I > understand correctly, the theory is that if all my NICs are on the > same interrupt, the kernel can stay in the interrupt handler (no > context switch) and service all the NICs at once, rather than handling > each separately. Am I understanding this right? Should I try to lump > all (or some) of my NICs onto the same IRQ? Or are there better > approaches (see below). i doubt that makes a difference these days. i don't worry any more. > Several sources have suggested using APIC, which should be available > in non-ancient hardware. I'm not sure if APIC replaces or complements > the suggestion above about interrupt sharing. I checked my box, and > my dmesg didn't mention APIC, so I don't think I'm taking advantage > of it right now. The -misc archives have oblique references to APIC > only being enabled on multiprocessor (MP) kernels rather than > uniprocessor (UP) ones. Is this still true? I also saw hints that > 4.6 now has APIC on in UP by default. Can anyone confirm or deny? 4.6 will just use the APIC. > Since PF isn't multi-core capable, I believed that UP was the way to > go for firewalls (and my machine isn't multicore anyway). However, > I'm happy to run MP if there are side benefits like APIC that would > increase performance. > > Next up, FreeBSD has been touting support for message-signaled > interrupts (MSI/MSI-X), claiming that this increases performance: > > http://onlamp.com/pub/a/bsd/2008/02/26/whats-new-in-freebsd-70.html?page=4 > > I'm not quite clear on whether this helps with a packet-forwarding > workload or not. Is there support for this in OpenBSD, or would it > not really help anyway? no support. > I've been getting errors like: > > WARNING: mclpool limit reached; increase kern.maxclusters > > So I did what it asked (I doubled the value to 12288), but am still > getting the error. I've heard of people increasing this much more > (20x the default!), but also taunts of insanity for doing so: > > http://monkey.org/openbsd/archive/misc/0407/msg01521.html > > So, what is a sane value for this? there is no easy or one-size-fits-all answer. > Next, I've seen interface drops (ifq.drops != 0), so I've cranked up > ifq.maxlen to 256 * #nics (1024) per recommendations on -misc. I > was still getting occasional drops, so I doubled to 2048, and am > holding steady there. I've seen recommendations not to go beyond > 2500; what should I be worried about in this case? High latency? > Memory issues? Do I really need to be worried about a few drops? latency mostly. memory isn't that much of an issue for this. i do have systems beyond 2500, but they handle many hundred MBit/s. > Finally, as was mentioned on the list a few days ago, increasing > recvspace/sendspace doesn't help with a firewall (except for > locally-sourced connections) because it's just forwarding packets. right. > Just so I'm totally clear, is this true even in the case of packet > reassembly (scrub) and randomization, or do those features cause the > firewall to terminate and re-initiate connections that would benefit > from the buffers? doesn't change a thing. send/recvspace only apply to sockets, aka stuff in userland. > For that matter, are there any protocol options that help performance > of a packet forwarding box (again, ignoring locally-sourced > connections)? I'm thinking about buffers, default MSS, ECN, window > scaling, SACK, etc. I know it doesn't hurt to turn them on, but am I > doing any good for the connections I'm forwarding? > > Thanks for any input and advice you can provide; I'm looking forward > to using PF for another 10 years... =) just use 4.6 and don't push buttons - you won't need to. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
