On 2/12/20 7:21 PM, Rudy wrote:
> I'm having issues with a box that is acting as a BGP router for my
network. 3 Chelsio cards, two T5 and one T6. It was working great
until I turned up our first port on the T6. It seems like traffic
passing in from a T5 card and out the T6 causes a really high load (and
high interrupts).
Looking better! I made some changes based on BSDRP which I hadn't known
about -- I think ifqmaxlen was the tunable I overlooked.
#
https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local
net.link.ifqmaxlen="16384"
Also, I ran chelsio_affinity to bind queues to specific CPU cores. The
script only supports a single t5 card, I am revising and will submit a
patch that will do multiple t5 and t6 cards.
I made both changes at once, and rebooted, so we'll never know which
fixed it. ;)
Right now, I have:
#/boot/loader.conf
#
# https://wiki.freebsd.org/10gFreeBSD/Router
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.holdoff_timer_idx=3
# Before FreeBSD 13, threading bad on router:
https://calomel.org/freebsd_network_tuning.html
machdep.hyperthreading_allowed="0"
hw.cxgbe.nrxq=16
hw.cxgbe.ntxq=16
hw.cxgbe.qsize_rxq=4096
hw.cxgbe.qsize_txq=4096
#hw.cxgbe.pause_settings="0"
#
https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local
net.link.ifqmaxlen="16384"
#/etc/sysctl.conf
# FRR needs big buffers for OSPF
kern.ipc.maxsockbuf=16777216
# Turn FEC off (doesn't work with Cogent)
dev.cc.0.fec=0
dev.cc.1.fec=0
# Entropy not from LAN ports... slows them down.
kern.random.harvest.mask=65551
net.inet.icmp.icmplim=400
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports
net.inet.tcp.msl=7500 # close lost tcp connections in 7.5
seconds (default 30)
net.inet.udp.blackhole=1 # drop any UDP packets to closed ports
#
hw.intr_storm_threshold=9000
net.inet.tcp.tso=0
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
