** Description changed: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. + There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d + claiming to resolve this error in some cases. It is already included in + the test kernel v5.15.0-41 and did not resolve the issue. - There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. + The following conditions are required to trigger the bug: + - A port VLAN must be assigned by the host + - The MTU must be set >1500 by the guest - These Virtual Machines do work with the Mainline v5.19 build on the host - and it includes the following two VIRTCHNL_OP_CONFIG_VSI_QUEUES related - commits that are not currently backported to v5.15 or any upstream - stable kernel: + It appears that this ice/iavf driver is quite immature as many + significant SR-IOV related fixes have landed in each of the recent + kernel releases and we may need to consider pro-actively backporting + more fixes. - 6096dae926a22e2892ef9169f582589c16d39639 ice: clear stale Tx queue settings before configuring [v5.18] - be2af71496a54a7195ac62caba6fab49cfe5006c ice: Fix queue config fail handling [v5.19] + [ Impact ] - Additionally during testing if we link down an interface and/or try to use netplan apply to start DHCP instead of manual configuration we triggered the following memory corruption bug: - efe41860008e57fb6b69855b4b93fdf34bc42798 ice: Fix memory corruption in VF driver [v5.19] + Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is + assigned by the host and the guest changes the MTU >1500. The guest iavf driver + logs the following error and traffic flow stops: + iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 + + There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used + without this patch. - It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. + [ Fix ] + + During queue configuration, VF's max packet size was set to + IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN + due to making some space for port VLAN as VF is not aware whether it's + in a port VLAN. This mismatch in sizes caused ice to reject queue + configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF + to VF with GET_VF_RESOURCES msg but VF does not look at this. + + In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu + received from pf with GET_VF_RESOURCES msg to make vf's + max_frame_size dependent from pf. Add check if received max_mtu is + not in eligible range then set it to IAVF_MAX_RXBUFFER. + + + [ Test Plan ] + + 1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port + vlan + + Openstack works, as does creating a VM directly with uvtool/libvirt. + + The syntax inside <interface> for the port VLAN is <vlan><tag + id=‘998’/></vlan> + + 2. Set an IP on the interface, set it’s link up and verify operation + + sudo ip a a 192.168.1.7/24 dev enp7s0 + sudo ip link set up dev enp7s0 + ping 192.168.1.1 + + 3. Set the MTU to 9000 + + sudo ip link set mtu 9000 dev enp7s0 + + 4. Test if the interface still works and whether the error was observed: + iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 + + + [ Where problems could occur ] + + The iavf driver is unified for multiple host drivers including ice and + i40e. + + A companion patch for i40e was also submitted as it did not set the max_mtu + although the ice driver already did. We should test the fixed iavf driver + against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that + patch. + + https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.ngu...@intel.com/ + + + [ Other Info ] + + Patch was developed by intel, has been accepted into v6.0-rc7 and already + released into upstream stable v5.4.215, v5.15.71 and v5.19.12. + + Early SRU is requested as the functionality is required now and there is no + workaround.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1983656 Title: SR-IOV VFs no traffic flow and error on Intel E810 (ice / iavf) Status in linux package in Ubuntu: Incomplete Bug description: Virtual Machines with SR-IOV VFs from an Intel E810-XXV [8086:159b] get no traffic flow and produce error messages in both the host and guest during network configuration. Environment: Ubuntu OpenStack Focal-Ussuri with OVN Host Kernel: v5.15.0-41-generic 20.04 Focal-HWE Guest Kernels: v5.4.x Focal, v5.15.0-41-generic Jammy Host Error Messages: ice 0000:98:00.1: VF 7 failed opcode 6, retval: -5 Guest Error Messages: iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 In the context of these errors "6" refers to the value of VIRTCHNL_OP_CONFIG_VSI_QUEUES It was found in these cases that the VM is able to successfully transmit packets but never receives any and the RX packet drop counters for the VF in "ip link" on the host increase equal to the RX packet count. There is a prior commit e6ba5273d4ede03d075d7a116b8edad1f6115f4d claiming to resolve this error in some cases. It is already included in the test kernel v5.15.0-41 and did not resolve the issue. The following conditions are required to trigger the bug: - A port VLAN must be assigned by the host - The MTU must be set >1500 by the guest It appears that this ice/iavf driver is quite immature as many significant SR-IOV related fixes have landed in each of the recent kernel releases and we may need to consider pro-actively backporting more fixes. [ Impact ] Intel E810 (host: ice) SR-IOV VFs (guest: iavf) fail when a port VLAN is assigned by the host and the guest changes the MTU >1500. The guest iavf driver logs the following error and traffic flow stops: iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 There is no workaround, Intel E810 SR-IOV VFs with MTU >1500 cannot be used without this patch. [ Fix ] During queue configuration, VF's max packet size was set to IAVF_MAX_RXBUFFER but on ice max frame size was smaller by VLAN_HLEN due to making some space for port VLAN as VF is not aware whether it's in a port VLAN. This mismatch in sizes caused ice to reject queue configuration with ERR_PARAM error. Proper max_mtu is sent from ice PF to VF with GET_VF_RESOURCES msg but VF does not look at this. In iavf change max_frame from IAVF_MAX_RXBUFFER to max_mtu received from pf with GET_VF_RESOURCES msg to make vf's max_frame_size dependent from pf. Add check if received max_mtu is not in eligible range then set it to IAVF_MAX_RXBUFFER. [ Test Plan ] 1. Create a focal VM and assign an Intel E810 (ice) SR-IOV VF with a port vlan Openstack works, as does creating a VM directly with uvtool/libvirt. The syntax inside <interface> for the port VLAN is <vlan><tag id=‘998’/></vlan> 2. Set an IP on the interface, set it’s link up and verify operation sudo ip a a 192.168.1.7/24 dev enp7s0 sudo ip link set up dev enp7s0 ping 192.168.1.1 3. Set the MTU to 9000 sudo ip link set mtu 9000 dev enp7s0 4. Test if the interface still works and whether the error was observed: iavf 0000:07:00.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6 [ Where problems could occur ] The iavf driver is unified for multiple host drivers including ice and i40e. A companion patch for i40e was also submitted as it did not set the max_mtu although the ice driver already did. We should test the fixed iavf driver against both i40e (Intel E7xx) and ice (Intel E8xx) cards with and without that patch. https://lore.kernel.org/netdev/20220919223428.572091-4-anthony.l.ngu...@intel.com/ [ Other Info ] Patch was developed by intel, has been accepted into v6.0-rc7 and already released into upstream stable v5.4.215, v5.15.71 and v5.19.12. Early SRU is requested as the functionality is required now and there is no workaround. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1983656/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
