Hey Jon from Nutanix AHV R&D here - We also see this issue on RHEL, SLE15, and vanilla kernels. In all three of those cases, the problem wasn't with AHV, but with a problematic set of backport that was tagged as a CVE, but caused a GSO-based performance regression. This can be reproduced on non-Nutanix AHV hosts too.
Cross-referencing: RHEL bugs: https://issues.redhat.com/browse/RHEL-62411 https://issues.redhat.com/browse/RHEL-67427 CentOS stream PR to fix this: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/5623 Generic reproduction from the Tailscale community, lot of great info in this thread: https://github.com/tailscale/tailscale/issues/13041#issuecomment-2294011607 Cross reference for SUSE SLES15 SP6: - broken in SLE15 SP6 with kernel 6.4.0-150600.23.22.1 (Sep-23-2024) - fixed in SLE15 SP6 with kernel 6.4.0-150600.23.25.1 (Oct-09-2024) - Versions are tracked here https://www.suse.com/support/kb/doc/?id=000019587 and aligned to the commit tags here: - bad commit: https://github.com/SUSE/kernel/commit/25229b4457540a2da59598839f39f8fbe3dfaa64 - good commit: https://github.com/SUSE/kernel/commit/e5010362f58efcc32c223cae45c7939ed3d04399 TLDR, the breaking commit is: net: drop bad gso csum_start and offset in virtio_net_hdr See commits with Fixes tag: Fixes: 89add40066f9 ("net: drop bad gso csum_start and offset in virtio_net_hdr") Fixing commits in our other cases with RHEL, SLE15, and generic vanilla upstream kernels: udp: fix receiving fraglist GSO packets net: tighten bad gso csum offset check in virtio_net_hdr Any chance this specific ubuntu kernel has "net: drop bad gso csum_start and offset in virtio_net_hdr" and not "net: tighten bad gso csum offset check in virtio_net_hdr"? ** Bug watch added: github.com/tailscale/tailscale/issues #13041 https://github.com/tailscale/tailscale/issues/13041 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2089012 Title: virtio-net slowdown and error message ens3: bad gso: type: 1, size: 1448 Status in linux package in Ubuntu: Confirmed Bug description: On Ubuntu Jammy, when upgrading from linux-image-5.15.0-124-generic to linux-image-5.15.0-125-generic, we notice on multiple Nutanix VMs: * A severe slowdown in network performance * A neverending stream of error messages in dmesg: [ 779.124760] ens3: bad gso: type: 1, size: 1398 [ 779.136866] ens3: bad gso: type: 1, size: 1398 [ 779.180993] ens3: bad gso: type: 1, size: 1448 [ 779.181002] ens3: bad gso: type: 1, size: 1448 [ 779.181005] ens3: bad gso: type: 1, size: 1448 [ 779.181073] ens3: bad gso: type: 1, size: 1448 [ 779.200088] ens3: bad gso: type: 1, size: 1448 [ 779.200096] ens3: bad gso: type: 1, size: 1448 [ 779.200098] ens3: bad gso: type: 1, size: 1448 [ 870.706518] net_ratelimit: 8 callbacks suppressed When downgrading to the old kernel, this problem disappears. Forcing an upgrade to linux-image-5.15.0-126-generic does not help. This looks a lot like https://bugzilla.kernel.org/show_bug.cgi?id=219129 but if I understand the bug tracker well, that patch has already been included in this kernel. Some details: $ lspci -v |grep -i net 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device $ lsb_release -rd Description: Ubuntu 22.04.5 LTS Release: 22.04 $ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.15.0.126.125 Candidate: 5.15.0.126.125 Version table: *** 5.15.0.126.125 500 500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages 100 /var/lib/dpkg/status 5.15.0.125.124 500 500 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages 5.15.0.25.27 500 500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages ProblemType: Bug DistroRelease: Ubuntu 22.04 Package: linux-image-5.15.0-125-generic 5.15.0-125.135 ProcVersionSignature: Ubuntu 5.15.0-125.135-generic 5.15.167 Uname: Linux 5.15.0-125-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Nov 19 12:49 seq crw-rw---- 1 root audio 116, 33 Nov 19 12:49 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu82.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A CasperMD5CheckResult: pass CloudArchitecture: x86_64 CloudID: none CloudName: none CloudPlatform: none CloudSubPlatform: config Date: Tue Nov 19 13:00:32 2024 InstallationDate: Installed on 2022-06-20 (882 days ago) InstallationMedia: Ubuntu-Server 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220421) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Lsusb-t: /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 480M MachineType: Nutanix AHV PciMultimedia: ProcEnviron: TERM=screen PATH=(custom, no user) LANG=C.UTF8 SHELL=/bin/bash ProcFB: 0 bochs-drmdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-125-generic root=UUID=2dabc077-db09-4f81-bbf2-b8390e5bc560 ro ipv6.disable=1 RelatedPackageVersions: linux-restricted-modules-5.15.0-125-generic N/A linux-backports-modules-5.15.0-125-generic N/A linux-firmware 20220329.git681281e4-0ubuntu3.35 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.release: 0.0 dmi.bios.vendor: SeaBIOS dmi.bios.version: nutanix-ahv-2.20220304.0.2619.el7 dmi.chassis.type: 1 dmi.chassis.vendor: Red Hat dmi.chassis.version: RHEL 7.6.0 PC (i440FX + PIIX, 1996) dmi.modalias: dmi:bvnSeaBIOS:bvrnutanix-ahv-2.20220304.0.2619.el7:bd04/01/2014:br0.0:svnNutanix:pnAHV:pvrRHEL7.6.0PC(i440FX+PIIX,1996):cvnRedHat:ct1:cvrRHEL7.6.0PC(i440FX+PIIX,1996):sku: dmi.product.family: Red Hat Enterprise Linux dmi.product.name: AHV dmi.product.version: RHEL 7.6.0 PC (i440FX + PIIX, 1996) dmi.sys.vendor: Nutanix To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2089012/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
