On 7/1/15 10:31 PM, Luigi Rizzo wrote:
On Wed, Jul 1, 2015 at 3:15 PM, Oliver Humpage <oli...@watershed.co.uk>
wrote:
Hello,
I hope this is a good list to post this on, I have a feeling the solution
is somewhere obscure in the networking layer.
I've set up an IPS system, using:
* FreeBSD 10.1 (guest OS, plenty of RAM/CPU)
* ESXi 5.5 (host OS, using Intel X520 10Gb cards. Not overloaded, all
graphs show it's got plenty of RAM/CPU spare at all times)
* vmxnet3 drivers
* ipfw (very small ruleset, basically just a divert rule)
* suricata, in ipfw divert mode
I'm having a couple of major issues.
The first is that every so often, even with relatively little traffic, the
load on the box suddenly spikes and pings to a neighbouring router (via the
divert rule) go from <1ms to >300ms. Generally this resolves itself after a
few minutes, although last night it went on for an hour until I restarted
ipfw and suricata.
typical problems with divert sockets are
1. if the interface has offloads (checksums, tso, etc.)
enabled, packets may be
moved up and down with invalid checksums resulting in
all sort of odd behaviours
2. likewise if there are huge segments (virtual nics often
pass up and down up to 64k in one shot) the divert socket
is unable to handle them and either truncates or drops
disabling all accelerations and setting the mtu to 1500 or so
usually fixes the above two.
3. divert probably loses important context on the packets
(e.g. incoming or outgoing interface) so when traffic is
reinjected bad things occur
actually this is one that it tries to keep. the interface name is in
the sockaddr (ipv4) hidden at the end.
If you use the same sockaddr that the packet arrived with, you will
pass the interface back again..
(actually in the man page I think).
4. performancewise, moving traffic back and forth to userspace
is a bad idea.
For the latter two, you might be better off using netmap
on vmxnet3 (in emulated mode, also disabling offloads),
and if i remember well a
couple of years ago there were
efforts to use
suricata on top of netmap.
Worst case, you can just use the netmap-enabled libpcap.
cheers
luigi
The second is that if I do a large download, eg a FreeBSD ISO, the download
usually hangs somewhere between 5MB and 100MB through. I can see traffic
trying to get through on neighbouring routers, it's just the interface with
the divert to suricata where they disappear into a black hole. The
connection speed is around 50Mb, btw.
Now it's possible it's suricata being weird, but there's nothing untoward
in its events and stats logs, and if I replay the traffic from a pcap file
then suricata processes everything fine (a pcap taken over a 90s period
during a slowdown is processed in under a second). So my guess is that if
suricata takes slightly longer than normal to process a packet, something
in the networking or ipfw divert system is tripping itself up. Maybe a
queue is filling up?
I've set net.inet.ip.fw.dyn_buckets=16384, and done an ipfw flush, but
net.inet.ip.fw.curr_dyn_buckets is stubbornly sticking at 256: have I done
something wrong? Other tunables I've set are:
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.point_to_point=0
kern.random.sys.harvest.interrupt=0
kern.ipc.soacceptqueue=1024
Can anyone suggest either tests to see what might be going wrong, or
tunables to help things run smoother? Both myself and a colleague have used
FreeBSD for over 15 years, and never quite seen anything like it.
Many thanks,
Oliver.
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
