Hi,thanks for your suggestion. Turns out, when i unplug the vlan interface from the bridge and put the ip address on the vlan interface, as you suggested, things start to work, e.g. arp resolves.
as soon as i put the ip and the vlan interface back on the bridge, things stop again. so where does this lead me? the problem is not in the vlan handling, but on the bridge?
I started playing with the net.link.bridge sysctls and indeed, when i set # sysctl net.link.bridge.inherit_mac=1 and then recreate the bridge # ifconfig bridge0 deletem ixl3.15 deletem vnet0.1 # ifconfig bridge0 addm ixl3.15 addm vnet0.1with the ip address still on bridge0 and ixl3, ixl3.15 and bridge0 all sharing the same mac address, arp starts resolving. but only for requests sent from the bridge0 interface. inside of the jail things still don't work (as the vnet interface again has another mac address).
# ifconfig ixl3ixl3: flags=28963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether a4:bf:01:76:ef:9d media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # ifconfig ixl3.15ixl3.15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP> ether a4:bf:01:76:ef:9d groups: vlan vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3 media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # ifconfig bridge0bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether a4:bf:01:76:ef:9d inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000 member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> # ping 192.168.55.1 PING 192.168.55.1 (192.168.55.1): 56 data bytes ^C --- 192.168.55.1 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss [! yes, the host does not answer on ICMP, but that is to be expected !] # arp -an ? (192.168.55.20) at a4:bf:01:76:ef:9d on bridge0 permanent [bridge]? (192.168.55.1) at b8:27:eb:47:8f:43 on bridge0 expires in 1197 seconds [bridge]
[...] [! into the jail !] JAIL # ifconfig epair0bepair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU> ether ac:16:2d:bd:b7:34 hwaddr 02:51:73:d1:33:0b inet 192.168.55.10 netmask 0xffffff00 broadcast 192.168.55.255 inet6 fe80::ae16:2dff:febd:b734%epair0b prefixlen 64 scopeid 0x2 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> JAIL # ping 192.168.55.1 PING 192.168.55.1 (192.168.55.1): 56 data bytes ^C --- 192.168.55.1 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss JAIL # arp -an ? (192.168.55.10) at ac:16:2d:bd:b7:34 on epair0b permanent [ethernet] ? (192.168.55.1) at (incomplete) on epair0b expired [ethernet]I conclude that there must be some mac address filtering going on in the data path, whether its on ixl or the bridge.
In dmesg I also see: >> bridge0: can't disable some capabilities on ixl3.15: 0x400but as of /usr/src/sys/net/if.h:233 this maps to IFCAP_LRO which afaik should not have any influence on L2 filtering.
Have to say, I'm out of ideas again. Never had something like this. So far just 'throwing interfaces on a bridge' worked in the past. Any ideas where to look next?
Thanks a lot & best Daniel On 1/25/24 08:22, Zhenlei Huang wrote:
I would encourage you to do plain VLAN tests, i.e. plug ixl3.15 out from bridge0On Jan 23, 2024, at 11:03 PM, Daniel <freebsd-...@c0decafe.de> wrote: Hi List,just recently I discovered a problem with the ixl(4) driver. Hopefully someone here can help me. my setup is as follows:Network ----- ixl3 interface ----- ixl3.15 vlan interface ----- bridge0 ----- vnet0.1 to jailthe problem now is that the jail can send data out (arp requests), i do see the responses on the ixl3 interface of the host, but they never make their way up to the ixl3.15 vlan interface (even though they are tagged correctly). To rule out that my config or the network is the cruel pit i did test the same setup with a cheap usb-ethernet adapter and there everything works as expected. I'm on FreeBSD 13.2-RELEASE-p8 and I did test both, the in kernel driver and the driver from ports intel-ixl-kmod-1.13.4_1.``` # ifconfig bridge0 deletem ixl3.15 # ifconfig bridge0 inet 192.168.55.20/24 delete ### to prevent confusion # ifconfig ixl3.15 inet 192.168.55.x/24 # ping -c1 192.168.55.1 ```May you please add the option -e to tcpdump, so that the link-level header can be printed out.Here is a bit of information on my environment: # uname -a FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64 # pciconf -lBbcevV pci0:25:0:3ixl3@pci0:25:0:3: class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x37d3 subvendor=0x8086 subdevice=0x35d5vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 10GbE SFP+' class = network subclass = ethernetbar [10] = type Prefetchable Memory, range 64, base 0xb0000000, size 16777216, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xb5000000, size 32768, enabledcap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 129 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO max read 512 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) cap 03[e0] = VPD ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 9aef76ffff01bfa4 ecap 000e[150] = ARI 1ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled0 VFs configured out of 32 supported First VF RID Offset 0x006d, VF RID Stride 0x0001 VF Device ID 0x37cdPage Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304ecap 0017[1a0] = TPH Requester 1ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable P2P Upstream Forwarding unavailable, P2P Egress Control unavailable P2P Direct Translated unavailable, Enhanced Capability unavailablePCI-e errors = Correctable Error Detected Unsupported Request Detected Corrected = Advisory Non-Fatal Error VPD ident = 'Example VPD' # ifconfig [...]ixl3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> ether a4:bf:01:76:ef:9d media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>ixl3.15: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP> ether a4:bf:01:76:ef:9d groups: vlan vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3 media: Ethernet autoselect (10Gbase-SR <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500ether 58:9c:fc:10:dd:05 inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000 member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 55 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> [...] # cat /etc/rc.conf [...] ifconfig_ixl3="up" vlans_ixl3="15" cloned_interfaces="bridge0" ifconfig_bridge0="addm ixl3.15 up" [...] # dmesg | grep ixlixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem 0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at device 0.0 numa-domain 0 on pci6ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared ixl0: Using 1024 TX descriptors and 1024 RX descriptors ixl0: Using 12 RX queues 12 TX queues ixl0: Using MSI-X interrupts with 13 vectors ixl0: Ethernet address: a4:bf:01:76:ef:9a ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 12/1024, RX 12/1024ixl1: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem 0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at device 0.1 numa-domain 0 on pci6ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared ixl1: Using 1024 TX descriptors and 1024 RX descriptors ixl1: Using 12 RX queues 12 TX queues ixl1: Using MSI-X interrupts with 13 vectors ixl1: Ethernet address: a4:bf:01:76:ef:9b ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active ixl1: SR-IOV ready ixl1: netmap queues/slots: TX 12/1024, RX 12/1024ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at device 0.2 numa-domain 0 on pci6ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C ixl2: Using 1024 TX descriptors and 1024 RX descriptors ixl2: Using 12 RX queues 12 TX queues ixl2: Using MSI-X interrupts with 13 vectors ixl2: Ethernet address: a4:bf:01:76:ef:9c ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5 ixl2: SR-IOV ready ixl2: netmap queues/slots: TX 12/1024, RX 12/1024ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem 0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at device 0.3 numa-domain 0 on pci6ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C ixl3: Using 1024 TX descriptors and 1024 RX descriptors ixl3: Using 12 RX queues 12 TX queues ixl3: Using MSI-X interrupts with 13 vectors ixl3: Ethernet address: a4:bf:01:76:ef:9d ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5 ixl3: SR-IOV ready ixl3: netmap queues/slots: TX 12/1024, RX 12/1024ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: Noneixl2: link state changed to UPixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: Noneixl3: link state changed to UP bridge0: can't disable some capabilities on ixl3.15: 0x400 ixl3: promiscuous mode enabled ixl3.15: promiscuous mode enabled from my packet traces: # tcpdump -vvv -i ixl3``` # tcpdump -nvei ixl3 ```tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture size 262144 bytes[...]13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28 13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46 13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28 13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46Here the answer can be see, its tagged with 802.1q tag 15 # tcpdump -vvv -i ixl3.15tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet), capture size 262144 bytes 14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28 14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28 14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28 14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.55.1 tell 192.168.55.10, length 28The answer cannot be seen on the VLAN interface ): I hope the list can help me out here, as I am lost. Thanks & best DanielBest regards, Zhenlei
