** Description changed:
- [Impact]
+ [ Impact ]
Users running VMs with `virtiofs` mounts where there is a page-size
mismatch between the host and guest kernels (e.g., a 4K page-size host
and a 64K page-size guest) experience "Cannot allocate memory"
(`ENOMEM`) errors when attempting to read directories.
This occurs because `virtiofsd` computes `max_pages` based on the host's
page size, but the guest converts `max_pages` back to bytes using its
own (larger) page size. As a result, the guest sends a `READDIR` request
that exceeds `virtiofsd`'s `MAX_BUFFER_SIZE`, causing the daemon to
reject the request.
The fix resolves this by capping the amount of directory data generated
locally to `MAX_BUFFER_SIZE` instead of rejecting the oversized request,
as `READDIR` is permitted to return fewer bytes than requested.
- [Test Plan]
+ [ Test Plan ]
- The Upstream MR has the details on how to reproduce and test the issue.
- Replicating it here:
+ # Set up repos and download packages
+ 0. Grab a fresh arm64 machine
+ 1. add deb-src to ubuntu.sources
+ 2. sudo apt build-dep linux
+ 3. sudo apt install virtme-ng qemu-system-arm
+ 4. sudo apt install virtiofsd # the package in question
+ # Set up vng and build kernel
+ 5. sudo usermod -aG kvm ubuntu && newgrp kvm # replace `ubuntu` with your user
+ 6. git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
+ 7. wait 10000 years; cd linux
+ 8. set up kernel config:
+ a. vng --kconfig
+ b. make menuconfig
+ c. kernel features -> page size -> set to 64k -> save -> exit
+ 9. vng --build
+
+ # Run the reproducer [from upstream merge request]
+ 10. getconf PAGE_SIZE # should print 4096 on host
+ 11. vng --run arch/arm64/boot/Image \
+ --memory 16G --cpus 4 --overlay-rwdir /tmp \
+ --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c
"import os; print(len(os.listdir(\"/tmp\")))"'
+
+ 12a. Unpatched package (fail):
+ ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
+ 7.1.0-rc7-virtme
+ 65536
+ ls: general io error: Cannot allocate memory (os error 12)
+ ls rc=1
+ Traceback (most recent call last):
+ File "<string>", line 1, in <module>
+ import os; print(len(os.listdir("/tmp")))
+ ~~~~~~~~~~^^^^^^^^
+ OSError: [Errno 12] Cannot allocate memory: '/tmp'
+
+ 12b. Patched package (success):
+ ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
+ 7.1.0-rc7-virtme
+ 65536
+ hsperfdata_root
+ snap-private-tmp
+ systemd-private-119639c5e17c430ca33b233f937d2924-ModemManager.service-EmreAg
+ systemd-private-119639c5e17c430ca33b233f937d2924-chrony.service-DJCwIj
+ systemd-private-119639c5e17c430ca33b233f937d2924-fwupd.service-DSIi3M
+ systemd-private-119639c5e17c430ca33b233f937d2924-polkit.service-w28tHD
+ systemd-private-119639c5e17c430ca33b233f937d2924-systemd-logind.service-bWi2S2
+ virtme_retvjz9uo9n
+ ls rc=0
+ 12
+
+ [ Other Info ]
+
+ Extra info from upstream MR:
---
The virtiofsd-side failure is easiest to reproduce after applying the
kernel patch that backs uncached readdir output with pages:
https://lore.kernel.org/all/[email protected]/
Without that kernel patch, the guest may fail earlier in the kernel before
the oversized READDIR request reaches virtiofsd.
- The reproducer uses virtme-ng/vng: https://github.com/arighi/virtme-ng
- Setup:
-
- - Host: arm64 4K page-size kernel
- - Guest: arm64 64K page-size kernel
- - virtiofsd: unpatched
- - vng: booting the guest with --overlay-rwdir /tmp
-
- On the host, from the kernel build tree containing the 64K guest Image:
-
- getconf PAGE_SIZE
-
- vng --run arch/arm64/boot/Image \
- --memory 16G --cpus 4 --overlay-rwdir /tmp \
- --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c
"import os; print(len(os.listdir(\"/tmp\")))"'
-
- Expected with unpatched virtiofsd:
-
- host PAGE_SIZE: 4096
- guest PAGE_SIZE: 65536
- ls: reading directory '/tmp': Cannot allocate memory
- ls rc=2
- OSError: [Errno 12] Cannot allocate memory: '/tmp'
-
- With this virtiofsd patch applied, the same command succeeds.
+ ...
+ (omitted for brevity)
+ ...
The page-size mismatch is what exposes the issue. virtiofsd computes
max_pages from the host page size, while the guest converts max_pages back
to bytes using the guest page size. With a 4K host and 64K guest, the guest
can send a READDIR size larger than virtiofsd's MAX_BUFFER_SIZE.
READDIR can return less than requested, so this patch caps the amount of
directory data generated locally instead of rejecting the request.
---
- [ Other Info ]
-
Target Releases: Ubuntu 24.04 and 26.04.
- Upstream Commit: d24cda8a325d server: do not reject oversized readdir
requests
+ Upstream Commit: d24cda8a325d server: do not reject oversized readdir requests
Upstream Link:
https://gitlab.com/virtio-fs/virtiofsd/-/commit/d24cda8a325d2a9ae1adc5acda57515ed2e8e1d2
Upstream Merge Request:
https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/316
** Also affects: rust-virtiofsd (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: rust-virtiofsd (Ubuntu Resolute)
Importance: Undecided
Status: New
** Also affects: rust-virtiofsd (Ubuntu Questing)
Importance: Undecided
Status: New
** Changed in: rust-virtiofsd (Ubuntu Resolute)
Assignee: (unassigned) => Alex Ramirez (kicchou)
** Changed in: rust-virtiofsd (Ubuntu Noble)
Assignee: (unassigned) => Alex Ramirez (kicchou)
** Changed in: rust-virtiofsd (Ubuntu Noble)
Status: New => In Progress
** Changed in: rust-virtiofsd (Ubuntu Resolute)
Status: New => In Progress
** Changed in: rust-virtiofsd (Ubuntu Questing)
Status: New => Won't Fix
** Changed in: rust-virtiofsd (Ubuntu)
Status: Confirmed => In Progress
** Description changed:
[ Impact ]
Users running VMs with `virtiofs` mounts where there is a page-size
mismatch between the host and guest kernels (e.g., a 4K page-size host
and a 64K page-size guest) experience "Cannot allocate memory"
(`ENOMEM`) errors when attempting to read directories.
This occurs because `virtiofsd` computes `max_pages` based on the host's
page size, but the guest converts `max_pages` back to bytes using its
own (larger) page size. As a result, the guest sends a `READDIR` request
that exceeds `virtiofsd`'s `MAX_BUFFER_SIZE`, causing the daemon to
reject the request.
The fix resolves this by capping the amount of directory data generated
locally to `MAX_BUFFER_SIZE` instead of rejecting the oversized request,
as `READDIR` is permitted to return fewer bytes than requested.
[ Test Plan ]
# Set up repos and download packages
0. Grab a fresh arm64 machine
1. add deb-src to ubuntu.sources
2. sudo apt build-dep linux
3. sudo apt install virtme-ng qemu-system-arm
4. sudo apt install virtiofsd # the package in question
# Set up vng and build kernel
5. sudo usermod -aG kvm ubuntu && newgrp kvm # replace `ubuntu` with your user
6. git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
7. wait 10000 years; cd linux
8. set up kernel config:
- a. vng --kconfig
- b. make menuconfig
- c. kernel features -> page size -> set to 64k -> save -> exit
+ a. vng --kconfig
+ b. make menuconfig
+ c. kernel features -> page size -> set to 64k -> save -> exit
9. vng --build
- # Run the reproducer [from upstream merge request]
+ # Run the reproducer (from upstream merge request)
10. getconf PAGE_SIZE # should print 4096 on host
11. vng --run arch/arm64/boot/Image \
- --memory 16G --cpus 4 --overlay-rwdir /tmp \
- --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c
"import os; print(len(os.listdir(\"/tmp\")))"'
+ --memory 16G --cpus 4 --overlay-rwdir /tmp \
+ --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c
"import os; print(len(os.listdir(\"/tmp\")))"'
12a. Unpatched package (fail):
ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
7.1.0-rc7-virtme
65536
ls: general io error: Cannot allocate memory (os error 12)
ls rc=1
Traceback (most recent call last):
- File "<string>", line 1, in <module>
- import os; print(len(os.listdir("/tmp")))
- ~~~~~~~~~~^^^^^^^^
+ File "<string>", line 1, in <module>
+ import os; print(len(os.listdir("/tmp")))
+ ~~~~~~~~~~^^^^^^^^
OSError: [Errno 12] Cannot allocate memory: '/tmp'
12b. Patched package (success):
ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
7.1.0-rc7-virtme
65536
hsperfdata_root
snap-private-tmp
systemd-private-119639c5e17c430ca33b233f937d2924-ModemManager.service-EmreAg
systemd-private-119639c5e17c430ca33b233f937d2924-chrony.service-DJCwIj
systemd-private-119639c5e17c430ca33b233f937d2924-fwupd.service-DSIi3M
systemd-private-119639c5e17c430ca33b233f937d2924-polkit.service-w28tHD
systemd-private-119639c5e17c430ca33b233f937d2924-systemd-logind.service-bWi2S2
virtme_retvjz9uo9n
ls rc=0
12
[ Other Info ]
Extra info from upstream MR:
---
The virtiofsd-side failure is easiest to reproduce after applying the
kernel patch that backs uncached readdir output with pages:
https://lore.kernel.org/all/[email protected]/
Without that kernel patch, the guest may fail earlier in the kernel before
the oversized READDIR request reaches virtiofsd.
...
(omitted for brevity)
...
The page-size mismatch is what exposes the issue. virtiofsd computes
max_pages from the host page size, while the guest converts max_pages back
to bytes using the guest page size. With a 4K host and 64K guest, the guest
can send a READDIR size larger than virtiofsd's MAX_BUFFER_SIZE.
READDIR can return less than requested, so this patch caps the amount of
directory data generated locally instead of rejecting the request.
---
Target Releases: Ubuntu 24.04 and 26.04.
Upstream Commit: d24cda8a325d server: do not reject oversized readdir requests
Upstream Link:
https://gitlab.com/virtio-fs/virtiofsd/-/commit/d24cda8a325d2a9ae1adc5acda57515ed2e8e1d2
Upstream Merge Request:
https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/316
** Description changed:
[ Impact ]
Users running VMs with `virtiofs` mounts where there is a page-size
mismatch between the host and guest kernels (e.g., a 4K page-size host
and a 64K page-size guest) experience "Cannot allocate memory"
(`ENOMEM`) errors when attempting to read directories.
This occurs because `virtiofsd` computes `max_pages` based on the host's
page size, but the guest converts `max_pages` back to bytes using its
own (larger) page size. As a result, the guest sends a `READDIR` request
that exceeds `virtiofsd`'s `MAX_BUFFER_SIZE`, causing the daemon to
reject the request.
The fix resolves this by capping the amount of directory data generated
locally to `MAX_BUFFER_SIZE` instead of rejecting the oversized request,
as `READDIR` is permitted to return fewer bytes than requested.
[ Test Plan ]
# Set up repos and download packages
0. Grab a fresh arm64 machine
1. add deb-src to ubuntu.sources
2. sudo apt build-dep linux
3. sudo apt install virtme-ng qemu-system-arm
4. sudo apt install virtiofsd # the package in question
# Set up vng and build kernel
5. sudo usermod -aG kvm ubuntu && newgrp kvm # replace `ubuntu` with your user
6. git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
7. wait 10000 years; cd linux
8. set up kernel config:
a. vng --kconfig
b. make menuconfig
c. kernel features -> page size -> set to 64k -> save -> exit
9. vng --build
# Run the reproducer (from upstream merge request)
10. getconf PAGE_SIZE # should print 4096 on host
11. vng --run arch/arm64/boot/Image \
--memory 16G --cpus 4 --overlay-rwdir /tmp \
--exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c
"import os; print(len(os.listdir(\"/tmp\")))"'
+ 12. There should be no errors in the output of 11.
- 12a. Unpatched package (fail):
+ 12a. Unpatched package output (fail):
ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
7.1.0-rc7-virtme
65536
ls: general io error: Cannot allocate memory (os error 12)
ls rc=1
Traceback (most recent call last):
File "<string>", line 1, in <module>
import os; print(len(os.listdir("/tmp")))
~~~~~~~~~~^^^^^^^^
OSError: [Errno 12] Cannot allocate memory: '/tmp'
- 12b. Patched package (success):
+ 12b. Patched package output (success):
ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4
--overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls
rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"'
7.1.0-rc7-virtme
65536
hsperfdata_root
snap-private-tmp
systemd-private-119639c5e17c430ca33b233f937d2924-ModemManager.service-EmreAg
systemd-private-119639c5e17c430ca33b233f937d2924-chrony.service-DJCwIj
systemd-private-119639c5e17c430ca33b233f937d2924-fwupd.service-DSIi3M
systemd-private-119639c5e17c430ca33b233f937d2924-polkit.service-w28tHD
systemd-private-119639c5e17c430ca33b233f937d2924-systemd-logind.service-bWi2S2
virtme_retvjz9uo9n
ls rc=0
12
[ Other Info ]
Extra info from upstream MR:
---
The virtiofsd-side failure is easiest to reproduce after applying the
kernel patch that backs uncached readdir output with pages:
https://lore.kernel.org/all/[email protected]/
Without that kernel patch, the guest may fail earlier in the kernel before
the oversized READDIR request reaches virtiofsd.
...
(omitted for brevity)
...
The page-size mismatch is what exposes the issue. virtiofsd computes
max_pages from the host page size, while the guest converts max_pages back
to bytes using the guest page size. With a 4K host and 64K guest, the guest
can send a READDIR size larger than virtiofsd's MAX_BUFFER_SIZE.
READDIR can return less than requested, so this patch caps the amount of
directory data generated locally instead of rejecting the request.
---
Target Releases: Ubuntu 24.04 and 26.04.
Upstream Commit: d24cda8a325d server: do not reject oversized readdir requests
Upstream Link:
https://gitlab.com/virtio-fs/virtiofsd/-/commit/d24cda8a325d2a9ae1adc5acda57515ed2e8e1d2
Upstream Merge Request:
https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/316
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2155048
Title:
[SRU] Backport request - server: do not reject oversized readdir
requests
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rust-virtiofsd/+bug/2155048/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
