> On Jan 23, 2024, at 11:03 PM, Daniel <[email protected]> wrote: > > Hi List, > > > just recently I discovered a problem with the ixl(4) driver. Hopefully > someone here can help me. my setup is as follows: > > > Network ----- ixl3 interface ----- ixl3.15 vlan interface ----- bridge0 ----- > vnet0.1 to jail > > > the problem now is that the jail can send data out (arp requests), i do see > the responses on the ixl3 interface of the host, but they never make their > way up to the ixl3.15 vlan interface (even though they are tagged correctly). > To rule out that my config or the network is the cruel pit i did test the > same setup with a cheap usb-ethernet adapter and there everything works as > expected. I'm on FreeBSD 13.2-RELEASE-p8 and I did test both, the in kernel > driver and the driver from ports intel-ixl-kmod-1.13.4_1. > I would encourage you to do plain VLAN tests, i.e. plug ixl3.15 out from bridge0
``` # ifconfig bridge0 deletem ixl3.15 # ifconfig bridge0 inet 192.168.55.20/24 delete ### to prevent confusion # ifconfig ixl3.15 inet 192.168.55.x/24 # ping -c1 192.168.55.1 ``` > > Here is a bit of information on my environment: > > # uname -a > FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC amd64 > > # pciconf -lBbcevV pci0:25:0:3 > ixl3@pci0:25:0:3: class=0x020000 rev=0x09 hdr=0x00 vendor=0x8086 > device=0x37d3 subvendor=0x8086 subdevice=0x35d5 > vendor = 'Intel Corporation' > device = 'Ethernet Connection X722 for 10GbE SFP+' > class = network > subclass = ethernet > bar [10] = type Prefetchable Memory, range 64, base 0xb0000000, size > 16777216, enabled > bar [1c] = type Prefetchable Memory, range 64, base 0xb5000000, size > 32768, enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 129 messages, enabled > Table in map 0x1c[0x0], PBA in map 0x1c[0x1000] > cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO > max read 512 > link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) > cap 03[e0] = VPD > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 9aef76ffff01bfa4 > ecap 000e[150] = ARI 1 > ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 32 supported > First VF RID Offset 0x006d, VF RID Stride 0x0001 > VF Device ID 0x37cd > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[1a0] = TPH Requester 1 > ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation > Blocking unavailable > P2P Req Redirect unavailable, P2P Cmpl Redirect > unavailable > P2P Upstream Forwarding unavailable, P2P Egress Control > unavailable > P2P Direct Translated unavailable, Enhanced Capability > unavailable > PCI-e errors = Correctable Error Detected > Unsupported Request Detected > Corrected = Advisory Non-Fatal Error > VPD ident = 'Example VPD' > > # ifconfig > [...] > ixl3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP> > ether a4:bf:01:76:ef:9d > media: Ethernet autoselect (10Gbase-SR <full-duplex>) > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > ixl3.15: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP> > ether a4:bf:01:76:ef:9d > groups: vlan > vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3 > media: Ethernet autoselect (10Gbase-SR <full-duplex>) > status: active > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > ether 58:9c:fc:10:dd:05 > inet 192.168.55.20 netmask 0xffffff00 broadcast 192.168.55.255 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 9 priority 128 path cost 2000 > member: ixl3.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 8 priority 128 path cost 55 > groups: bridge > nd6 options=9<PERFORMNUD,IFDISABLED> > [...] > > > > > # cat /etc/rc.conf > [...] > ifconfig_ixl3="up" > vlans_ixl3="15" > cloned_interfaces="bridge0" > ifconfig_bridge0="addm ixl3.15 up" > [...] > > > > # dmesg | grep ixl > ixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem > 0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at device 0.0 numa-domain 0 on > pci6 > ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 > ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared > ixl0: Using 1024 TX descriptors and 1024 RX descriptors > ixl0: Using 12 RX queues 12 TX queues > ixl0: Using MSI-X interrupts with 13 vectors > ixl0: Ethernet address: a4:bf:01:76:ef:9a > ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active > ixl0: SR-IOV ready > ixl0: netmap queues/slots: TX 12/1024, RX 12/1024 > ixl1: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.3-k> mem > 0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at device 0.1 numa-domain 0 on > pci6 > ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 > ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO shared > ixl1: Using 1024 TX descriptors and 1024 RX descriptors > ixl1: Using 12 RX queues 12 TX queues > ixl1: Using MSI-X interrupts with 13 vectors > ixl1: Ethernet address: a4:bf:01:76:ef:9b > ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active > ixl1: SR-IOV ready > ixl1: netmap queues/slots: TX 12/1024, RX 12/1024 > ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem > 0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at device 0.2 numa-domain 0 on > pci6 > ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 > ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C > ixl2: Using 1024 TX descriptors and 1024 RX descriptors > ixl2: Using 12 RX queues 12 TX queues > ixl2: Using MSI-X interrupts with 13 vectors > ixl2: Ethernet address: a4:bf:01:76:ef:9c > ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active > ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5 > ixl2: SR-IOV ready > ixl2: netmap queues/slots: TX 12/1024, RX 12/1024 > ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.3-k> mem > 0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at device 0.3 numa-domain 0 on > pci6 > ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0 > ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C > ixl3: Using 1024 TX descriptors and 1024 RX descriptors > ixl3: Using 12 RX queues 12 TX queues > ixl3: Using MSI-X interrupts with 13 vectors > ixl3: Ethernet address: a4:bf:01:76:ef:9d > ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active > ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5 > ixl3: SR-IOV ready > ixl3: netmap queues/slots: TX 12/1024, RX 12/1024 > ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: > None, Autoneg: False, Flow Control: None > ixl2: link state changed to UP > ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: > None, Autoneg: False, Flow Control: None > ixl3: link state changed to UP > bridge0: can't disable some capabilities on ixl3.15: 0x400 > ixl3: promiscuous mode enabled > ixl3.15: promiscuous mode enabled > > > from my packet traces: > > # tcpdump -vvv -i ixl3 > May you please add the option -e to tcpdump, so that the link-level header can be printed out. ``` # tcpdump -nvei ixl3 ``` > tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture size 262144 > bytes > [...] > 13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > 13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at > b8:27:eb:47:8f:43 (oui Unknown), length 46 > 13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > 13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.55.1 is-at > b8:27:eb:47:8f:43 (oui Unknown), length 46 > > Here the answer can be see, its tagged with 802.1q tag 15 > > > # tcpdump -vvv -i ixl3.15 > tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet), capture size > 262144 bytes > 14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > 14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > 14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > 14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has > 192.168.55.1 tell 192.168.55.10, length 28 > > The answer cannot be seen on the VLAN interface ): > > > I hope the list can help me out here, as I am lost. > > > Thanks & best > > > > Daniel > Best regards, Zhenlei
