On 2/12/20 7:21 PM, Rudy wrote:
I'm having issues with a box that is acting as a BGP router for my
network. 3 Chelsio cards, two T5 and one T6. It was working great
until I turned up our first port on the T6. It seems like traffic
passing in from a T5 card and out the T6 causes a really high load
(and high interrupts).
Update, I have found the chelsio_affinity script and run that, doesn't
seem to help. I did notice the T6 card is on numa-domain 0 and the T5
cards on on numa-domain 1. Does that matter?
Here is the relevant part of dmesg:
t6nex0: <Chelsio T62100-LP-CR> mem
0xc5300000-0xc537ffff,0xc4000000-0xc4ffffff,0xc5984000-0xc5985fff irq 48
at device 0.4 numa-domain 0 on pci12
cc0: <port 0> numa-domain 0 on t6nex0
cc1: <port 1> numa-domain 0 on t6nex0
t6nex0: PCIe gen3 x16, 2 ports, 34 MSI-X interrupts, 66 eq, 33 iq
t5nex0: <Chelsio T540-LP-CR> mem
0xed300000-0xed37ffff,0xec000000-0xecffffff,0xed984000-0xed985fff irq 88
at device 0.4 numa-domain 1 on pci18
cxl0: <port 0> numa-domain 1 on t5nex0
cxl1: <port 1> numa-domain 1 on t5nex0
cxl2: <port 2> numa-domain 1 on t5nex0
cxl3: <port 3> numa-domain 1 on t5nex0
t5nex0: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq
t5nex1: <Chelsio T540-LP-CR> mem
0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xfb984000-0xfb985fff irq 98
at device 0.4 numa-domain 1 on pci20
cxl4: <port 0> numa-domain 1 on t5nex1
cxl5: <port 1> numa-domain 1 on t5nex1
cxl6: <port 2> numa-domain 1 on t5nex1
cxl7: <port 3> numa-domain 1 on t5nex1
t5nex1: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq
Box does have 20 cores and hyper-threading is disabled.
sysctl hw.ncpu
hw.ncpu: 20
I've added updated /boot/loader.conf to the following:
hw.cxgbe.nrxq=16
hw.cxgbe.ntxq=16
hw.cxgbe.qsize_rxq=4096
hw.cxgbe.qsize_txq=4096
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.holdoff_timer_idx=3
machdep.hyperthreading_allowed="0"
Box works fine unless I turn on TX on the cc1 device (I am filtering
routes learned from that interface via BGP).
Rudy
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
