Re: ix0 and ix1 ifconfig options different on Supermicro board

BulkMailForRudy Wed, 27 Nov 2019 09:51:23 -0800


Thank you, Vincenzo.


To anyone else looking this up, 'iperf' is not dependent on the TSO/LRO.
  iperf -c 10.1.1.1 -P 4   --->  9.4Gbps
  iperf3 -c 10.1.1.1 -P 4  --->  5.1Gbps

I still find it odd that the default FreeBSD install has differentoptions for ix0 and ix1.


Rudy



On 11/23/19 12:51 AM, Vincenzo Maffione wrote:

Hi,
   TSO/LRO (for IPv4 and/or IPv6) will increase TCP bulk throughput on
machine X for those TCP connection where X is one of the two endpoints,
that is TCP connections that are local to X. That's why you are seing iperf
achieving higher throughput with TSO/LRO enabled.
TSO means that your local TCP stack will pass down large (e.g. 32K) packets
to the NIC driver, and the NIC will take care of segmentation. This is
beneficial for two reasons: (1) the segmentation work is done in hardware
rather than in the CPU, and this is typically faster (and also, you save
the CPU time for other stuff); (2) the per-packet cost of protocol
processing (TCP, IP, Ethernet) is amortized over a large amount of bytes,
which means that your total per-byte CPU time will be way lower. Most of
the gain actually comes from (2).
LRO is similar, but in the receive direction.

However, if your device is a router it means that it forwards packets.
Therefore the local TCP stack is not involved, so TSO simply does not apply
(at least in FreeBSD).
I think LRO applies, but there is a latency hit, as suggested by the wiki
page you pointed.

So no, enabling TSO/LRO will not increase the forwarding rate, but possibly
increase latency. You should keep it disabled.

Cheers,
   Vincenzo

Il giorno ven 22 nov 2019 alle ore 22:47 BulkMailForRudy <
cra...@monkeybrains.net> ha scritto:

I just did another test to a machine with a Chelsio card.

   Server D (cxl3) -> Server A = 3.5Gbps

Turning on flags lro tso4 tso6 vlanhwtso , yields

   Server D (cxl3) -> Server A = 9.1 Gbps

Oddly, this was an ipv4 iperf, but tso6 seems to help.

I had settings turned off per
https://wiki.freebsd.org/10gFreeBSD/Router#Disabling_LRO_and_TSO

Servers A,B, and C are all running services.  Server D is acting as a
router.  Are the LRO and TSO only for TCP to the box, or will it
increase speeds for forwarding if I enable it?


Thanks,

Rudy


On 11/22/19 1:30 PM, BulkMailForRudy wrote:

I have nearly identical setups, but ix0 and ix1 are getting different
options at boot.  This seems to be the only difference I see between
machines and I am trying to answer the question, Why can Server A
iperf close to line rate while the other servers can not?

The Test:  iperf -P 3 -c REMOTE_ADDR

Server A (ix1) -> Server C (ix0)  = 9.4Gbps
Server B (ix0)-> Server C (ix0) = 5.6Gbps
Server C (ix0)-> A (ix1) or B (ix0)  = 5.0Gbps


The motherboards are identical between A,B and C and the configs very
similar.  The only difference is that Server A is plugged into ix1
while Server B and C are using ix0.


I am not modifying the flags at boot (eg ifconfig -tso), yet ix0 lacks
TXCSUM,TSO4,TSO6,LRO,WOL.

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500

options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>

         ether *ac:1f:6b:6a:14:6*4
         media: Ethernet autoselect (10Gbase-T <full-duplex>)
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

         ether *ac:1f:6b:6a:14:6*5
         media: Ethernet autoselect (10Gbase-T <full-duplex>)

I did try adding some flags to ix0 and -- not sure if this was the
reason -- the box started acting oddly and I ended up rebooting it.


My hunch has is that there is somethign with the TSO4.


Rudy

_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: ix0 and ix1 ifconfig options different on Supermicro board

Reply via email to