https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235607
Bug ID: 235607
Summary: Incorrect checksums with NAT on vtnet with offloading
Product: Base System
Version: 12.0-STABLE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: b...@freebsd.org
Reporter: sjorge+sig...@blackdot.be
### description
The issue only pops up when there is no valid tcp checksum present on the
source traffic, it 'works' when the csum is valid. I did verify with the
illumos people about if a bad checksum will be pass along as is or not with
offloading enabled:
[17:03:15] <sjorge> rzezeski qq, if a guest with csum enabled sends a
checksummed packet does it get discarded and recalculate when it hits the
physical nic before it goes on the wire? Or is it kept as is?
[17:03:26] <sjorge> From my captures it seems to be kept as is?
[17:05:26] <rzezeski> sjorge: It depends on the guest OS. In illumos, either
the IP stack calcs the checksum on Tx or places a flag on the dblk to have it
calculated by hardware. If you do both then the hardware will calculate a
second sum over the current one and it will be incorrect.
[17:05:42] <rzezeski> 1) <reply to diff question> 2) I have no idea how
FBSD/Linux work in that regard, but they would have the same issue to deal
with.
I beleive the later case is what is happening here, pf nat adds checksum based
on empty initial checksum and then it gets pass as-is out the nic. That would
also explain why a packet with a valid checksum that comes in will pass back
out correctly after pf nat.
<-> = phsyical link, aka cat5/cat6 plugged into a NIC or Switch
<~> = loopback link, aka traffic between bhyve guests, between host and bhyve
guest, ... stuff that never hit the MAC layer.
<.> = wireless link
This flow that is currently broken (only when pf nat is involved):
fbsd_guest1 <~> fbsd_guest_fw <-> switch(1) <-> modem
win10_guest <~> fbsd_guest_fw <-> switch(1) <-> modem
This flow is OK:
macbook <-> switch <->(2) fbsd_guest_fw <-> switch <-> modem
macbook <.> AP <-> switch <-> fbsd_guest_fw <-> switch <-> modem
1) using port-mirror on the switch I was able to confirm packets with a bad
checksum end up on the wire
2) the bhyve guest has a vnic on the physical nic (they are considered bridge
for the host OS)
#### workaround but comes at a performance hit
root@nattest:~ # ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -lro
root@nattest:~ # ifconfig vtnet1 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -lro
Re-enabeling these on will make the behavior return, this can be done live!
### uname -a
FreeBSD nattest 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64
### ifconfig output
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=6c05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:22:06:05:01:0a
inet 192.168.0.212 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet 10Gbase-T <full-duplex>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=6c05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:22:06:0a:01:01
inet 10.23.10.87 netmask 0xffffff00 broadcast 10.23.10.255
media: Ethernet 10Gbase-T <full-duplex>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
groups: pflog
### /etc/pf.conf
if_wan="vtnet0"
if_lan="vtnet1"
net_lan=$if_lan:network
scrub on lo0 all random-id
scrub on vtnet0 all random-id
scrub on vtnet1 all random-id
nat on $if_wan inet from $net_lan to any -> ($if_wan) port 1024:65535
antispoof log for vtnet0
antispoof log for vtnet1
block in all
pass on lo0 all
pass inet proto icmp all icmp-type { echoreq, unreach } keep state
pass out all keep state
pass in on $if_lan proto tcp from $net_lan to any port { 22, 80, 443 }
### /etc/rc.conf
hostname="nattest"
gateway_enable="YES"
ifconfig_vtnet0="DHCP"
ifconfig_vtnet1="10.23.10.87 netmask 255.255.255.0"
zfs_enable="YES"
clear_tmp_enable="YES"
sshd_enable="YES"
dumpdev="AUTO"
pf_enable="YES"
pflog_enable="YES"
--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
