Hi,
I'm trying to troubleshoot some performance issues for high speed data
transfers across a long network path with a fairly high bandwidth delay.
The router is running OpenBSD 4.6 - admittedly well overdue for an
update to 5.0, but I did briefly test 5.0 on a backup machine and saw
much the same. The hardware is a Dell 1750 (getting old but seemed to be
keeping up) with Intel PRO/1000MT interfaces (82546GB) inside and out.
It's been in use for a number of years, passing 750-800Mbps traffic
quite happily.
sysctl tuning is limited to raising net.inet.ip.maxlen from 256 -> 1024.
We now have a need for high bandwidth data transfers to a Physics lab
across the world, and with some help from our network group have a route
set up which can pass 700Mbps - but only when our local endpoint is
outside the OpenBSD router.
Doing local tests with iperf using linux boxes as the endpoints we can
pass up to 750Mbps with ~0.3% UDP packet loss, whether that traffic goes
through the router or not. The OpenBSD machine itself shows no packet
drops with "netstat -id" or "sysctl net.inet.ip.drops", and no Ierrs on
the network interfaces.
However long-distance iperf tests to the distant lab show very different
results...
through router: 10/80 Mbps (in/out)
bypassing router: 400/750 Mbps (in/out)
so clearly the router is affecting *something*. All the data suggests
it's not dropping packets. Could it be interfering with window scaling?
There is a pf ruleset in place, of course, but since stateful tracking
became the default years ago (at which time I removed most of the "keep
state" bits), it should be hard to misconfigure that...? There's nothing
very exotic in pf.conf - default deny and a bunch of "pass quick" rules
for various services; no NAT-type rules. Tried disabling packet
scrubbing on a whim, but no change.
One interesting point is that for the 750Mbps local iperf tests through
the router I do see fairly high interrupt load (~50%), which I'd
expect... but for the very poor long-distance iperf tests I see very low
load.
Both the linux endpoints have been tuned for the high BDP (the network
latency is around 250ms - so huge tcp buffers to cover the in-flight
data), but none of these values are relevant for packet forwarding on
the router...
I'm lost for where to look next, and would appreciate any ideas!
Graham
--
