I'm in a similar situation, 3ghz single cpu on a supermicro board with
two syskonnect cards, previously running around 50,000 packets the cpu
would bounce between 20 - 80 cpu utilized. Switched to the mp kernel,
I'm now topping out at 215,000 showing basically no utilization.
mike blodgett
CSL, UW-Madison CS Dept.
dormando wrote:
So,
My latest update;
Theo mentioned the single CPU kernels don't make use of APIC interrupt
controllers, just ISA. I booted my single P4 systems into the bsd.mp
kernel, and behold there's a major difference in speed!
Now the systems no longer claim 95%+ CPU held in interrupts, but claim
to be 100% idle most of the time, bouncing into 1-6% sys CPU every few
seconds, and holding at 0% int CPU. Traffic changed from lossy at 120
megs, to maxed out at 150 megabits, ~70k pps per interface.
At that point traffic very obviously flatlined, but it did not dip or
fail. I saw no visible CPU load, interrupts were around 7.8k/sec per
active NIC. It looked almost like I had set an altq limit of 150
megabits. Any idea on how to profile where my packets are spending
most of their time? I'm not so great with this level of
troubleshooting, but I would love to get better at it.
Right now I have two machines in a semi-carp cluster. A 3.7 stable
box, and a -current as of oct 15th. 3.7 doesn't have the tuner Henning
mentioned, but 3.8 and -current do. Set net.inet.ip.ifq.maxlen=250 on
the -current box and traffic went up to 160 megabits and flatlined
again.
The next thing I'm trying tomorrow morning is switching the internal
interface to one of the bge nics. The systems have two bge nics
built-in, and one PCI-X 133mhz intel dual port 1000MT server nic.
Right now the int/ext are on the intel card and the pfsync int is on
bge1.
-Dormando
On 10/19/05, Henning Brauer <[EMAIL PROTECTED]> wrote:
eh, this is really only good for benching, because otherwise we stop
traversing the pf ruleset for very short amounts of time if we are
about to exhaust CPU. this allows already established connections to
live on and the OP to log in to the box via console and take
countermeasures. if you already ahd an ssh sessionto teh box it has
good chances to survive and you can even take countermeasures over that.
what you really want to do for high speed routers is increasing
net.inet.ip.ifq.maxlen
I currently use 250 on some routers which seems good, but I need to do
more tests before I can make qualified assumptions about good values.
This is the max length of a queue in the input path, and the default of
50 packets is too small for high speed routers with modern GigE cards
that can put about that into teh queue with one single int. Or even more.
In the end I think we need a better default based on some factors like
ip forwarding enabled and summarized link speed and RAM in teh box or
somesuch. Ryan and I discussed that on the ferry earlier this year and
have some good ideas, now we just need some time to work on it ;(
* Schvberle Daniel <[EMAIL PROTECTED]> [2005-10-18 18:36]:
Hi,
I was trying to bench routing pps with pf on and henning gave me
some advice which I think might help you too. For my benching purposes
it helped break the 200k pps barrier with current but no guaranties
that it'll do you any good or that it won't hurt you.
<quote>
The high drop rates
are a anti-DDoS measure - yeah, that pretty much makes benching
impossible...
you could change IF_INPUT_ENQUEUE in sys/net/if.h so that it looks like
#define IF_INPUT_ENQUEUE(ifq, m) { \
if (IF_QFULL(ifq)) { \
IF_DROP(ifq); \
m_freem(m); \
} else \
IF_ENQUEUE(ifq, m); \
}
i. e. remove these two lines:
if (!(ifq)->ifq_congestion) \
if_congestion(ifq); \
that means the congestion flag will never be set.
or you add a return; as first statement in if_congestion() in if.c.
<endquote>
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of dormando
Sent: Monday, October 17, 2005 8:29 PM
To: misc@openbsd.org
Subject: Very high interrupts on a supermicro machine.
Hey all,
Attached is a dmesg of one of a pair of supermicro based firewalls I
recently bought. I had set them up as a CARP/pfsync redundant pair of
frontend firewalls for our network. However, after they reached 15,000
interrupts per second (~ 110 megabits of our site traffic),
they passed 90%
CPU usage through interrupts and stopped being useful.
The machines have two built-in BGE nics. I swapped in an
Intel PRO/1000MT
Dual Port Server Nic into a PCI-X 133mhz PCI slot, but it
made absolutely no
difference in the interrupt load. The current firewalls in
place are freebsd
machines running on supermicro hardware with two em based
built-in nics
running past 40k interrupts without passing 50% CPU load on
interrupts. The
only error I can see in the dmesg was this:
pcibios0: no compatible PCI ICU found: ICU vendor 0x8086
product 0x2640
pcibios0: Warning, unable to fix up PCI interrupt routing
pcibios0: PCI bus #5 is the last bus
... which as far as I can read, is "harmless", but potentially causing
higher interrupt load?
Any hints as to where I should look next would be great. I'm about to
install the latest -current snapshot on the machine to see if
there's a
recent fix.
I'm about 95% sure this is the motherboard we're using:
http://www.supermicro.com/products/motherboard/P4/E7221/P8SCT.
cfm I'll check
with the order guy and confirm the PO.
There's a 3.4ghz P4 CPU in it, the two built-in nics, and a
single PCI-X
133mhz PCI port which I used for the dual port server nic
from intel. SATA
harddrive for what it's worth. Running OpenBSD 3.7 as a PF
firewall. I've
tried changing a bunch of BIOS options, disabling interrupts,
etc. I haven't
compiled my own kernel or built the OS or anything.
Thanks,
-Dormando
--
BS Web Services, http://www.bsws.de/
OpenBSD-based Webhosting, Mail Services, Managed Servers, ...
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)
