In other words, until I see like 100% system usage in one core, I
would have room to grow?
Probably not. Mutexes in FreeBSD use "adaptive spinning". This means
that when a thread is unable to acquire a mutex, if the owner of the
mutex is still running on another CPU core then the thread will spin and
wait for the mutex to become free rather than block. This improves the
latency of acquiring the mutex and saves us many expensive calls through
the scheduler, but the downside is that you have one heavily contended
mutex you can have many cores wasted spinning on the lock. In your case
it is likely that your interrupt threads are spinning on the pf lock.
You can partially confirm this by profiling the system. The quick way is:
Truly fascinating!
So my cores that are sucking up CPU on interrupts may be just "spinning"
waiting for a lock and not doing real work? That would explain a lot.
kldload hwpmc
pmcstat -S unhalted-cycles -T
(if unhalted-cycles does not work, try instructions).
We'll try this. It doesn't look like the hwpmc module exists on this
box but we'll bring it in and try that.
If _mtx_lock_sleep is the top function then you are hitting mutex
contention and more cores are unlikely to help. In this case you might
actually be able to improve performance by *reducing* the number of
threads that are contending in pf, for example by using fewer receive
queues in your NIC.
If this machine is dedicated for pf then setting sysctl net.isr.direct=0
might also improve performance, by forcing all packets to go through a
single netisr thread (assuming that net.isr.maxthreads is 1). Note that
this will apply to traffic that does not go through pf, so if this
machine were doing other network things (e.g. serving NFS) then it would
bottleneck your other traffic behind your pf traffic.
It's only doing pf stuff so this is something we'll try. Thanks for the
enlightening information!!
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
