On Fri, Apr 26, 2013 at 10:49 AM, Erich Weiler <wei...@soe.ucsc.edu> wrote:
> The pf isn't a process, so you can't see it in top. pf has some helper >> threads however, but packet processing isn't performed by any of them. >> > > But the work pf does would show up in 'system' on top right? So if I see > all my CPUs tied up 100% in 'interrupts' and very little 'system', would it > be a reasonable assumption to think that if I got more CPU cores to handle > the interrupts that eventually I would see 'system' load increase as the > interrupt load became faster to be handled? And thus increase my bandwidth? > > In other words, until I see like 100% system usage in one core, I would > have room to grow? > Probably not. Mutexes in FreeBSD use "adaptive spinning". This means that when a thread is unable to acquire a mutex, if the owner of the mutex is still running on another CPU core then the thread will spin and wait for the mutex to become free rather than block. This improves the latency of acquiring the mutex and saves us many expensive calls through the scheduler, but the downside is that you have one heavily contended mutex you can have many cores wasted spinning on the lock. In your case it is likely that your interrupt threads are spinning on the pf lock. You can partially confirm this by profiling the system. The quick way is: kldload hwpmc pmcstat -S unhalted-cycles -T (if unhalted-cycles does not work, try instructions). If _mtx_lock_sleep is the top function then you are hitting mutex contention and more cores are unlikely to help. In this case you might actually be able to improve performance by *reducing* the number of threads that are contending in pf, for example by using fewer receive queues in your NIC. If this machine is dedicated for pf then setting sysctl net.isr.direct=0 might also improve performance, by forcing all packets to go through a single netisr thread (assuming that net.isr.maxthreads is 1). Note that this will apply to traffic that does not go through pf, so if this machine were doing other network things (e.g. serving NFS) then it would bottleneck your other traffic behind your pf traffic. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
