Adding all newer (supported) releases to track the necessary follow-up changes (string parsing fix and lower default max pool size)
Libvirt AppArmor change is here: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2170732 ** Also affects: qemu (Ubuntu Stonking) Importance: Undecided Status: New ** Also affects: qemu (Ubuntu Tenacious) Importance: High Assignee: Alex Ramirez (kicchou) Status: Fix Released ** Also affects: qemu (Ubuntu Resolute) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu Tenacious) Status: Fix Released => In Progress ** Changed in: qemu (Ubuntu Noble) Status: New => Confirmed ** Changed in: qemu (Ubuntu Resolute) Status: New => Confirmed ** Changed in: qemu (Ubuntu Stonking) Status: New => Confirmed ** Changed in: qemu (Ubuntu Tenacious) Status: In Progress => Confirmed ** Changed in: qemu (Ubuntu Noble) Importance: Undecided => High ** Changed in: qemu (Ubuntu Tenacious) Importance: High => Low ** Changed in: qemu (Ubuntu Resolute) Importance: Undecided => Low ** Changed in: qemu (Ubuntu Stonking) Importance: Undecided => Low ** Changed in: qemu (Ubuntu Tenacious) Milestone: noble-updates => None ** Changed in: qemu (Ubuntu Noble) Milestone: None => noble-updates ** Changed in: qemu (Ubuntu Resolute) Milestone: None => resolute-updates ** Changed in: qemu (Ubuntu Stonking) Milestone: None => stonking-updates ** Changed in: qemu (Ubuntu Tenacious) Milestone: None => ubuntu-27.04 ** Changed in: qemu (Ubuntu Noble) Status: Confirmed => In Progress ** Changed in: qemu (Ubuntu Resolute) Assignee: (unassigned) => Alex Ramirez (kicchou) ** Changed in: qemu (Ubuntu Stonking) Assignee: (unassigned) => Alex Ramirez (kicchou) ** Description changed: [ Impact ] QEMU 8.2.2's coroutine pool implementation can cause the QEMU process's VMA count to hit the Linux vm.max_map_count limit (`65530` in Jammy, `1048576` in Noble), causing QEMU to abort with "failed to allocate memory for stack" or "failed to set up stack guard page" during coroutine creation. This eventually causes running guests to crash. A VM with 32 vCPUs and two virtio-blk disks gets a theoretical maximum VMA mappings of 1122816, higher than the Noble kernel default. See the model in section {1} of 'Other info' for detail on how we calculate this number. In practice the number of coroutines and thus VMAs that will actually be allocated is typically much smaller since the total number of allocated coroutines depends on concurrent IO, thus forcing an increase in the size of the thread-local caches. Many small IOs all at once drain both the thread-local and global pools, causing more coroutines to be allocated. Coroutine growth after VM start is hypothesized to behave logarithmically. This behavior was introduced in upstream 4c41c69e05fe28c (v7.0.0-rc0) and therefore only affects Ubuntu 24.04. The reporter experienced crashes while running Noble's QEMU in a container on a 5.15 kernel (with the old `vm.max_map_count=65530`). Even with a relatively modest VM configuration (36 vCPUs, 1-3 disks), we’ve observed numbers of cached VMAs in the 30,000 - 70,000 range. Crashes are expected to be significantly less likely with Noble's sysctl config since reaching its default vm.max_map_count of 1048576 would require a significant fraction of the total allocated coroutines to be active concurrently (or a 10x increase in the number of vCPUs). Crashes with the default Noble configuration have not been reported, but are certainly possible with large guests. While crashes can be worked around by adjusting the sysctl, this is still a resource leak that can leave a large number of unused coroutines allocated but unusable. 86a637e4810 allows for significantly better reuse of those cached idle coroutines (2,000 - 5,000 as opposed to the 30,000 - 70,000 cited above). This new coroutine caching implementation allows for drastically larger VMs for Noble users in the near future (think 100+ vCPUs, multiple disks, and IOThreads) without needing to change the 1048576 VMA limit from the default in that release. For instance, current ARM CPUs have 192 cores on a single package, and dual socket systems can have 384 CPUs. It's not infeasible that core count could double within 5 years, and Noble will still be supported (through ESM) by then. Users with such systems will likely want to dedicate 100+ vCPUs to a single guest. The drastically improved caching should mean that no Noble user would need to consider increasing the default 1048576 vm.max_map_count limit with the caching change per our current understanding of how coroutines are cached. [ Test Plan ] 1. Run the upstream iotests (in a virtual machine) to exercise coroutines (not all of these tests are run during package build): ```sh gbp pq switch sudo apt build-dep ./ sudo apt install seabios ipxe-qemu make -f debian/rules configure-qemu # Binaries needed for iotests ninja -C b/qemu modules make -f debian/rules build-x86-optionrom cp -a b/optionrom/*.bin /usr/share/seabios/* /usr/lib/ipxe/qemu/* b/qemu/qemu-bundle/usr/share/qemu/ make -C b/qemu/ check-block cd b/qemu/ ./tests/qemu-iotests/check -qcow2 # Expected failures 151 281 307 graph-changes-while-io ``` 2. Run the Ubuntu Server team's qemu-migration-test [1] 3. The following procedure reproduces mmap exhaustion: ``` lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=8GiB n0 ``` In the VM: ```sh sudo apt install libvirt-daemon-system virtinst wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-server-cloudimg-amd64.img sudo cp noble-server-cloudimg-amd64.img /var/lib/libvirt/images/testvm-root.qcow2 sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data2.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data3.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data4.raw 10G cat > user-data <<EOF #cloud-config password: ubuntu chpasswd: expire: false ssh_pwauth: true package_update: true packages: - fio EOF sudo virt-install \ -n testvm \ --os-variant=ubuntu24.04 \ --ram=4096 --vcpus=255 \ --import \ --disk path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \ --disk path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data2.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data3.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data4.raw,bus=virtio,cache=none,driver.io=io_uring \ --graphics none \ --network network=default \ --cloud-init user-data=user-data # Ctrl+Shift+a+] sudo virsh shutdown testvm ``` Limit write throughput in the block layer so that writes allocate a coroutine but aren't able to complete right away: ```sh for disk in vdb vdc vdd vde; do sudo virsh blkdeviotune testvm "${disk}" --total-iops-sec 2000 --current done ``` Use `sudo virsh edit testvm` to pin each disk to its own iothread; otherwise all IO will occur on the main thread. Add `iothread='1'` to the `<driver>` section of each 'testvm-data' disk, incrementing the number for each disk. ```sh # To observe the crash, reduce the max map count to the Jammy value sudo sysctl -w vm.max_map_count=65530 sudo virsh start testvm sudo virsh console testvm ``` On the host, you can monitor the maps with: ```sh pid=$(pidof qemu-system-x86_64) while sleep 1; do sudo grep -c '' "/proc/${pid}/maps"; done ``` In the testvm: ```sh while true; do - sudo fio \ - --ioengine=io_uring \ - --direct=1 \ - --runtime=10 \ - --time_based=1 \ - --rw=randread \ - --bs=4k \ - --iodepth=128 \ - --numjobs=256 \ - --group_reporting \ - --name=vdb --filename=/dev/vdb \ - --name=vdc --filename=/dev/vdc \ - --name=vdd --filename=/dev/vdd \ - --name=vde --filename=/dev/vde + sudo fio \ + --ioengine=io_uring \ + --direct=1 \ + --runtime=10 \ + --time_based=1 \ + --rw=randread \ + --bs=4k \ + --iodepth=128 \ + --numjobs=256 \ + --group_reporting \ + --name=vdb --filename=/dev/vdb \ + --name=vdc --filename=/dev/vdc \ + --name=vdd --filename=/dev/vdd \ + --name=vde --filename=/dev/vde done ``` Patched behavior: /proc/pid/maps levels off and stays consistent through the fio run. (With these parameters we observed approximately TBD). Unpatched behavior: /proc/pid/maps increases with each fio run until the VM crashes with one of: - failed to allocate memory for stack: Cannot allocate memory - failed to set up stack guard page: Cannot allocate memory With a higher vm.max_map_count and the parameters above, `wc -l /proc/$qemu_pid/maps` has been observed at 509362. 4. Performance regression test (results should be provided both before & after): ```sh for bs in 4k 128k; do for iodepth in 1 8 16 32 64; do sudo fio --output-format=json \ --output=fio-$bs-$iodepth.json \ --ioengine=libaio \ --direct=1 \ --numjobs=$(nproc) \ --runtime=30 \ --ramp_time=5 \ --time_based=1 \ --rw=randread \ --bs=$bs \ --iodepth=$iodepth \ --name=job1 \ --filename=/dev/vdb done done ``` [1] https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu- migration-test/+git/qemu-migration-test [ Where problems could occur ] The patches restructure the way coroutines are allocated and deallocated. Coroutines are used extensively in the QMP/HMP protocol implementation, block drivers and live migrations; this change is somewhat fundamental to QEMU's coroutine infrastructure. If the change were wrong/broken, it could cause: - failing block device drivers, causing failed boots or broken I/O operations within affected guests - incorrect/strange behavior when interacting with a QEMU process over QMP/HMP - failed/broken live migrations - performance regressions as the patch introduces a lock in the coroutine pool hot path (coroutine_pool_get) Upstream has two follow-ups to the refactor, 9352f80cd926fe2dde7c89b93ee33bb0356ff40e and 25bc7d16fa96b0ff881c83ed225ea380fe427c78. 9352f80cd will be included in the upload; 25bc7d16fa fixes a false-positive compiler warning. 86a637e4 parses the contents of `/proc/sys/vm/max_map_count` incorrectly [1]. This means that without Hector's linked follow-up, the patch removes any limitation on the maximum size of the coroutine idle pool. The introduction of a read() from procfs will introduce an apparmor denial for VMs using libvirt's AppArmor security driver [2]. If AppArmor is active for a VM, this change will remove any limitation of the maximum size of the coroutine idle pool. If [1] and [2] are also applied, workloads which require many coroutines may experience a performance regression (as the patches set a hard limit on the size of the coroutine pool, so coroutines needed above the pool size are allocated/deallocated on demand). The coroutine pool size is a function of `vm.max_map_count`, so increasing the max map count is the recommended workaround: ``` sysctl -w vm.max_map_count=1048576 ``` It's likely that workloads running into this performance regression would also have been affected by the original bug. [1] https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html [2] https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 [ Other Info ] {1} We can model the coroutine pool size as a function of VM properties. There are 3 kinds of threads which run coroutines (have non-empty thread-local pools): the main thread, vCPU threads, and iothreads. In (unpatched) QEMU 8.2.2 the theoretical maximum of idle coroutines in each thread-local pool is given by: (see hw/block/virtio-blk.c:1638) pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × queue_size_i / 2) Defaults: queue_size_i = 256 (hw/block/virtio-blk.c:1722) num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) The global pool is twice as large as `pool_max_size` (util/qemu- coroutine.c:102); since thread-local pools steal the global pool, they may have at most 2 × pool_max_size coroutines: theoretical maximum mappings = 2 × (vCPUs + iothreads + 2) × 2 × pool_max_size Noble's QEMU 8.2.2 keeps a thread-local list of coroutines `alloc_pool` (util/qemu-coroutine.c:40); this list can grow to thousands or even tens of thousands of coroutines _for each thread_. Each coroutine has two VMAs created with mmap(2). If there are enough threads available to the QEMU process, the number of coroutines may cause the total number of mappings to exceed `vm.max_map_count`, causing the QEMU process to fail to allocate memory and crash. - - - - {2} QEMU's coroutines are a userspace threading implementation. They are used frequently in QEMU's virtual disk drivers. Each coroutine creates two anonymous mappings with mmap(2). When a coroutine terminates it is cached in a global "release pool"; once that is full (2 x max pool size), terminated coroutines are cached in a thread-local "alloc pool" (up to the max pool size, per thread). When the thread-local pool is empty, the global pool is moved to a thread- local pool. Pooled coroutines are only freed once both pools are full. From upstream, this bug occurs because "per-thread pools can grow to tens of thousands of coroutines. Each coroutine causes 2 virtual memory areas to be created. Eventually vm.max_map_count is reached and memory- related syscalls fail. The per-thread pool sizes are non-uniform and depend on past coroutine usage in each thread, so it's possible for one thread to have a large pool while another thread's pool is empty." The new approach "does not leave large numbers of coroutines pooled in a thread that may not use them again. In order to perform well it amortizes the cost of global pool accesses by working in batches of coroutines instead of individual coroutines." - - - - {3} + The follow-up AppArmor change to libvirt is here: + https://bugs.launchpad.net/bugs/2170732 + Upstream commit (coroutine: cap per-thread local pool size): https://gitlab.com/qemu-project/qemu/-/commit/86a637e4 Upstream commit (coroutine: reserve 5,000 mappings): https://gitlab.com/qemu-project/qemu/-/commit/9352f80c Upstream commit (util/coroutine: fix -Werror=maybe-uninitialized false-positive): https://gitlab.com/qemu-project/qemu/-/commit/25bc7d RHEL9 carries these two patches as 176 and 177 respectively (86a637e4 and 9352f80c). They are present in the CentOS Stream 9 repositories. CentOS Stream 9 Patch 176 (coroutine: cap per-thread local pool size): https://gitlab.com/redhat/centos-stream/src/qemu-kvm/-/commit/98ea1c9ee22d75dbe4f5060ccdf8164418ffe44b CentOS Stream 9 Patch 177 (coroutine: reserve 5,000 mappings): https://gitlab.com/redhat/centos-stream/src/qemu-kvm/-/commit/dd639ab6336c9bae2a40fe857036a432e0077319 Additionally, the following bugs were identified in 86a637e4 which require follow-ups for the patch to work as intended: - Missing AppArmor rule: https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 + Missing AppArmor rule (tracked in LP: #2170732): https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 Incorrect string parsing: https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html ** Changed in: qemu (Ubuntu Noble) Assignee: (unassigned) => Wesley Hershberger (whershberger) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2168803 Title: QEMU 8.2.2 guests crash by exhausting vm.max_map_count with coroutines To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2168803/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
