After multiple tests and tweaks i believe the issue is not with the HW or Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out properly for the speed that the HW can do, i keep getting widely varying results when testing. Below is an example of two tests, with about a 15 second break in between the two.
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 991 MBytes 8.32 Gbits/sec 268 579 KBytes [ 5] 1.00-2.00 sec 945 MBytes 7.93 Gbits/sec 369 777 KBytes [ 5] 2.00-3.00 sec 793 MBytes 6.65 Gbits/sec 60 1.03 MBytes [ 5] 3.00-4.00 sec 666 MBytes 5.59 Gbits/sec 203 976 KBytes [ 5] 4.00-5.01 sec 575 MBytes 4.78 Gbits/sec 202 1.13 MBytes [ 5] 5.01-6.00 sec 169 MBytes 1.43 Gbits/sec 134 699 KBytes [ 5] 6.00-7.00 sec 1.21 GBytes 10.4 Gbits/sec 383 1.08 MBytes [ 5] 7.00-8.00 sec 1.21 GBytes 10.4 Gbits/sec 0 1.16 MBytes [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 124 780 KBytes [ 5] 9.00-10.00 sec 690 MBytes 5.79 Gbits/sec 316 605 KBytes [ 5] 10.00-11.00 sec 685 MBytes 5.75 Gbits/sec 97 854 KBytes [ 5] 11.00-12.00 sec 1.08 GBytes 9.30 Gbits/sec 383 538 KBytes [ 5] 12.00-13.00 sec 682 MBytes 5.72 Gbits/sec 88 870 KBytes [ 5] 13.00-14.00 sec 678 MBytes 5.69 Gbits/sec 123 964 KBytes [ 5] 14.00-15.00 sec 670 MBytes 5.62 Gbits/sec 290 763 KBytes [ 5] 15.00-16.00 sec 1.01 GBytes 8.71 Gbits/sec 228 1.08 MBytes [ 5] 16.00-17.00 sec 886 MBytes 7.44 Gbits/sec 118 615 KBytes [ 5] 17.00-18.00 sec 734 MBytes 6.16 Gbits/sec 291 902 KBytes [ 5] 18.00-19.00 sec 1.04 GBytes 8.96 Gbits/sec 212 323 KBytes [ 5] 19.00-20.00 sec 710 MBytes 5.96 Gbits/sec 193 547 KBytes [ 5] 20.00-21.00 sec 693 MBytes 5.82 Gbits/sec 370 942 KBytes [ 5] 21.00-22.00 sec 704 MBytes 5.91 Gbits/sec 80 1022 KBytes [ 5] 22.00-23.00 sec 1.26 GBytes 10.8 Gbits/sec 262 965 KBytes [ 5] 23.00-24.00 sec 828 MBytes 6.94 Gbits/sec 202 763 KBytes [ 5] 24.00-25.00 sec 774 MBytes 6.49 Gbits/sec 227 581 KBytes [ 5] 25.00-26.00 sec 734 MBytes 6.15 Gbits/sec 256 664 KBytes [ 5] 26.00-27.00 sec 753 MBytes 6.32 Gbits/sec 331 540 KBytes [ 5] 27.00-28.00 sec 764 MBytes 6.41 Gbits/sec 298 823 KBytes [ 5] 28.00-29.00 sec 757 MBytes 6.35 Gbits/sec 123 850 KBytes [ 5] 29.00-30.00 sec 754 MBytes 6.32 Gbits/sec 74 970 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 24.4 GBytes 6.98 Gbits/sec 6305 sender [ 5] 0.00-30.00 sec 24.4 GBytes 6.98 Gbits/sec receiver iperf Done. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.81 GBytes 15.5 Gbits/sec 0 1.11 MBytes [ 5] 1.00-2.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 MBytes [ 5] 2.00-3.00 sec 1.98 GBytes 17.0 Gbits/sec 0 1.11 MBytes [ 5] 3.00-4.00 sec 2.11 GBytes 18.1 Gbits/sec 0 1.11 MBytes [ 5] 4.00-5.00 sec 2.12 GBytes 18.2 Gbits/sec 0 1.11 MBytes [ 5] 5.00-6.00 sec 2.16 GBytes 18.5 Gbits/sec 0 1.11 MBytes [ 5] 6.00-7.00 sec 1.90 GBytes 16.3 Gbits/sec 0 1.12 MBytes [ 5] 7.00-8.02 sec 1.28 GBytes 10.8 Gbits/sec 0 1.12 MBytes [ 5] 8.02-9.00 sec 1.83 GBytes 16.0 Gbits/sec 0 1.17 MBytes [ 5] 9.00-10.00 sec 1.91 GBytes 16.4 Gbits/sec 0 1.20 MBytes [ 5] 10.00-11.00 sec 1.79 GBytes 15.3 Gbits/sec 0 1.60 MBytes [ 5] 11.00-12.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.60 MBytes [ 5] 12.00-13.00 sec 1.69 GBytes 14.5 Gbits/sec 0 1.61 MBytes [ 5] 13.00-14.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.61 MBytes [ 5] 14.00-15.00 sec 1.60 GBytes 13.8 Gbits/sec 0 1.61 MBytes [ 5] 15.00-16.00 sec 1.89 GBytes 16.2 Gbits/sec 0 1.61 MBytes [ 5] 16.00-17.00 sec 1.76 GBytes 15.1 Gbits/sec 0 1.61 MBytes [ 5] 17.00-18.00 sec 1.93 GBytes 16.6 Gbits/sec 0 1.61 MBytes [ 5] 18.00-19.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.61 MBytes [ 5] 19.00-20.00 sec 1.68 GBytes 14.5 Gbits/sec 0 1.61 MBytes [ 5] 20.00-21.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 MBytes [ 5] 21.00-22.00 sec 1.75 GBytes 15.1 Gbits/sec 0 1.61 MBytes [ 5] 22.00-23.00 sec 1.76 GBytes 15.1 Gbits/sec 0 1.61 MBytes [ 5] 23.00-24.00 sec 2.05 GBytes 17.7 Gbits/sec 0 1.61 MBytes [ 5] 24.00-25.00 sec 1.79 GBytes 15.4 Gbits/sec 0 1.61 MBytes [ 5] 25.00-26.00 sec 1.71 GBytes 14.7 Gbits/sec 0 1.61 MBytes [ 5] 26.00-27.00 sec 2.08 GBytes 17.9 Gbits/sec 0 1.61 MBytes [ 5] 27.00-28.00 sec 2.09 GBytes 18.0 Gbits/sec 0 1.61 MBytes [ 5] 28.00-29.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.61 MBytes [ 5] 29.00-30.00 sec 1.81 GBytes 15.6 Gbits/sec 0 1.61 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 55.2 GBytes 15.8 Gbits/sec 0 sender [ 5] 0.00-30.00 sec 55.2 GBytes 15.8 Gbits/sec receiver iperf Done.
