Hello Tim, I tested the new Version Kernel on a VM in Markus's Cluster: root@hostname:~# uname -ar Linux hostname 6.8.0-53-generic #54 SMP PREEMPT_DYNAMIC Fri Feb 21 02:43:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@hostname:~# wget -O /dev/null https://fsn1-speed.hetzner.com/10G.bin --2025-02-21 08:51:20-- https://fsn1-speed.hetzner.com/10G.bin Auflösen des Hostnamens fsn1-speed.hetzner.com (fsn1-speed.hetzner.com) … 78.46.170.2, 2a01:4f8:0:a232::2 Verbindungsaufbau zu fsn1-speed.hetzner.com (fsn1-speed.hetzner.com)|78.46.170.2|:443 … verbunden. HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK Länge: 10737418240 (10G) [application/octet-stream] Wird in ‘/dev/null’ gespeichert. /dev/null 14%[==================> ] 1.45G 108MB/s ETA 81s It's downloading with full speed, no limitations. It seems to be working now. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2098961 Title: Severe network performance issues with Kernel 6.8.0-53 and 6.11.0-17 (Xubuntu) on Proxmox 8.3 (bnxt_en + VirtIO) Status in linux package in Ubuntu: Triaged Bug description: Since upgrading to Kernel 6.8.0-53 and 6.11.0-17 on Xubuntu 22.04/24.04 VMs, we are experiencing extremely low network speeds (100-200 KB/s) when running these VMs in a Proxmox 8.3.4 cluster with AMD CPUs and Broadcom NetXtreme-E network adapters (bnxt_en driver). With Kernel 6.8.0-31 inside the same Xubuntu VM, everything works as expected. Other VMs (Debian, Ubuntu Server) running on the same Proxmox cluster are not affected. The same Xubuntu VM, when moved to a different Proxmox cluster (Intel CPUs, Intel NICs, ZFS storage), runs fine with Kernel 6.8.0-53 and 6.11.0-17. The issue only affects Xubuntu VMs running on a Proxmox 8.3.4 cluster with AMD CPUs, Broadcom NICs, and Ceph storage. Expected behavior: Network performance should be normal (10 Gbps or more) without issues. Actual behavior: Xubuntu VMs with Kernel 6.8.0-53 and 6.11.0-17 experience extremely slow network speeds (100-200 KB/s). iperf3 tests show very low bandwidth (5-10 Mbps). Disabling GSO, TSO, and GRO (ethtool -K ens18 tso off gso off gro off) did not improve performance. Other VMs (Debian, Ubuntu Server) on the same cluster do not have this issue. The issue disappears when booting Kernel 6.8.0-31 inside the same Xubuntu VM. Steps to reproduce: Proxmox environment: Proxmox 8.3.4 (Kernel 6.8.12-8-pve) AMD CPU + Broadcom NetXtreme-E BCM57504 (bnxt_en driver) + Ceph storage VMs running Xubuntu 22.04 or 24.04 with VirtIO network (virtio-net) Inside the Xubuntu VM: Install and boot Kernel 6.8.0-53 or Kernel 6.11.0-17 Test network performance: wget -O /dev/null http://speed.hetzner.de/1GB.bin iperf3 -c <server-ip> -P 4 → Speed is extremely slow (~200 KB/s - 5 Mbps). Reboot the VM with Kernel 6.8.0-31 sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-31-generic" sudo reboot → Network performance returns to normal (10 Gbps+). Hardware & environment details: Proxmox cluster (affected environment): Proxmox 8.3.4 (Kernel 6.8.12-8-pve) AMD CPUs Broadcom NetXtreme-E BCM57504 25G SFP28 Ethernet (bnxt_en driver) Ceph storage backend VMs use VirtIO network adapter (virtio-net) Bridging on Broadcom NICs Working cluster (same Xubuntu VM, no issue): Proxmox 8.3.4 (Kernel 6.8.12-8-pve) Intel CPUs Intel I350 NICs (igb driver) ZFS storage backend Same Xubuntu VM works fine with Kernel 6.8.0-53 and 6.11.0-17 Debugging information: Xubuntu VM Kernel versions (affected): uname -r 6.8.0-53-generic 6.11.0-17-generic Broadcom firmware version (Proxmox host): ethtool -i enp67s0f0np0 | grep firmware firmware-version: 229.0.154.0/pkg 229.1.123.0 dmesg output related to bnxt_en NICs (Proxmox host): dmesg | grep bnxt [ 2.324347] bnxt_en 0000:c5:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 60 seconds, may trigger hung task watchdog [ 2.364280] bnxt_en 0000:c5:00.0 eth0: Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem b7030000, node addr 14:23:f2:5d:18:14 ... → No critical errors, but timeout warnings and performance degradation. Possible causes: 🔹 Regression in virtio-net or bnxt_en driver in Kernel 6.8.0-53 and 6.11.0-17 🔹 Interaction issue between virtio-net and bnxt_en on AMD + Broadcom hardware 🔹 Proxmox bridge settings on Broadcom NICs interacting with Xubuntu's Kernel 6.8.0-53 and 6.11.0-17 Next Steps & Further Tests Since this is a production system with 200+ running VMs, we will conduct further tests once we schedule maintenance. Planned additional tests: Test a fresh Ubuntu Server VM with Kernel 6.8.0-53 and 6.11.0-17 (to rule out Xubuntu-specific issues). Verify bnxt_en and virtio-net driver versions in Kernel 6.8.0-31 vs. 6.8.0-53 and 6.11.0-17. Check IRQ assignments and CPU affinity for VirtIO network interfaces. 📌 If this issue is already reported, please link to the relevant bug report. 📌 Otherwise, we are happy to provide further debugging information. Summary: Xubuntu VMs running Kernel 6.8.0-53 and 6.11.0-17 on a Proxmox 8.3.4 cluster with AMD CPUs + Broadcom NICs suffer from extreme network slowdowns (~200 KB/s). Booting the same VM with Kernel 6.8.0-31 restores full performance. Same Kernels work fine on Intel-based Proxmox clusters. Disabling TSO, GSO, GRO does NOT fix the issue. Likely a regression in virtio-net or bnxt_en. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: linux-image-6.8.0-53-generic 6.8.0-53.55 ProcVersionSignature: Ubuntu 6.8.0-53.55-generic 6.8.12 Uname: Linux 6.8.0-53-generic x86_64 ApportVersion: 2.28.1-0ubuntu3.1 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/seq: sysadmin 1219 F.... pipewire CRDA: N/A CasperMD5CheckResult: pass Date: Thu Feb 20 11:52:51 2025 InstallationDate: Installed on 2025-02-20 (0 days ago) InstallationMedia: Xubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827) IwConfig: lo no wireless extensions. ens18 no wireless extensions. Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU Tablet Lsusb-t: /: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=uhci_hcd/2p, 12M |__ Port 001: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 12M MachineType: QEMU Standard PC (i440FX + PIIX, 1996) ProcFB: 0 bochs-drmdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.8.0-53-generic root=UUID=5f2acbcf-06d2-464a-83a0-12ef507709a5 ro quiet splash vt.handoff=7 PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon. RelatedPackageVersions: linux-restricted-modules-6.8.0-53-generic N/A linux-backports-modules-6.8.0-53-generic N/A linux-firmware 20240318.git3b128b60-0ubuntu2.9 RfKill: SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.release: 0.0 dmi.bios.vendor: SeaBIOS dmi.bios.version: rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-9.0 dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org:bd04/01/2014:br0.0:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-9.0:cvnQEMU:ct1:cvrpc-i440fx-9.0:sku: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-9.0 dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098961/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
