On 08/07/2013 12:55 PM, Maxim Khitrov wrote:
I found a number of recommendations for the things to keep an eye on,
but nothing that gave me any ideas on what else to try for improving
the performance. Specifically, I looked at netstat -m on both systems,
where everything was well below the limits (<500 mbufs in use during
the test). I see about 8200/8700 (ix0/total) interrupts in systat with
1500 MTU. CPU usage in top is split between two cores, one at ~80%
interrupt and the other at ~80% system. Most of the time all four
cores are at least 10% idle (hyper-threading is disabled in BIOS).
netstat -i shows no errors for ix0 and sysctl net.inet.ip.ifq.drops is
at 0 on both systems.
First, 80% interrupt time says that there's really no more time to process
packets. AFAIK, OpenBSD only processes interrupts on CPU0.
If 127.0.0.1 only got 12Gb/S, that strongly suggests that it
can't process fast enough to handle a real 10Gb/S load.
Second, 10Gb/S is the hardware bit rate. The only way to see that is to send
one infinite-length packet. Every packet has sync, preamble, and trailer
bit times
which reduce the time for sending data payload.
Third, does your test program account for IP and TCP/UDP headers? If it
doesn't,
then that's even more overhead deducted from the available payload.
Fourth, can the interface card really send at line rate? Many 100K and 1G
cards can't. I've never used a 10G card.
Fifth, if your test program segfaults at any time, it can't be trusted.
Sixth, there were out-of-order packets. This could mean packet loss, either
because either sender or receiver cards were confused/overloaded/something
or the receiver CPU went deaf long enough to drop packets.
I used to work on high performance networks (at least, high performance
for the time). All of the above were real problems I encountered.
Geoff Steckel
