On 2025-05-02 14:13, Fiona Ebner wrote:
Am 02.05.25 um 13:52 schrieb Markus Frank:
On 2025-04-28 12:17, Fiona Ebner wrote:
As reported in the community forum [0] and the virtio-win project [1],
virtiofsd will run into its open file limit when used with a Windows
guest that reads too many files. It's also reported that the issue
does not occur with Linux guests and a workaround is using
'--inode-file-handles=mandatory' on virtiofsd command line.
I was able to reproduce the issue with a directory containing a million
files.
The virtiofs share became unusable when I tried to list the files with
'ls'.
This patch allows to list all the files without problems.
The option is described as follows in the vritiofsd help:
typo: virtiofsd
Will fix!
When to use file handles to reference inodes instead of O_PATH file
descriptors (never, prefer, mandatory)
and the default is 'never'.
Fix the above issue by using 'prefer' rather than 'mandatory', because
that should not break other edge cases:
prefer: Attempt to generate file handles, but fall back to O_PATH
file descriptors where the underlying filesystem does not support
file handles. Useful when there are various different filesystems
under the shared directory and some of them do not support file
handles.
[0]: https://forum.proxmox.com/threads/165565/
[1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
Signed-off-by: Fiona Ebner <f.eb...@proxmox.com>
Tested-by: Markus Frank <m.fr...@proxmox.com>
---
Didn't get around to measure the performance impact yet, so feel free
to check that if you test this patch. While not being broken is more
important than good performance, it would still be good to know for
completeness.
I made a few read/write tests with fio in a Windows 11 guest.
Since the option affects file handles, it would be more interesting to
test something that involves handling many files rather than just IO.
Okay, here is a test that creates 10000 4kB files.
without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share
--announce-submounts --syslog):
PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k
--directory="testdir" --ioengine=windowsaio --create_on_open=1
--file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 4 (f=18): [_(3),f(1),_(2),f(1),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8000: Fri May 2 15:45:44 2025
write: IOPS=967, BW=3870KiB/s (3962kB/s)(114MiB/30049msec); 0 zone resets
slat (nsec): min=0, max=632300, avg=20663.88, stdev=13715.30
clat (usec): min=494, max=12174, avg=2302.34, stdev=676.20
lat (usec): min=512, max=12195, avg=2323.00, stdev=675.62
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1352], 10.00th=[ 1532], 20.00th=[ 1745],
| 30.00th=[ 1926], 40.00th=[ 2073], 50.00th=[ 2212], 60.00th=[ 2376],
| 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3195], 95.00th=[ 3523],
| 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5211], 99.95th=[ 5669],
| 99.99th=[ 8455]
bw ( KiB/s): min= 1621, max= 4387, per=99.24%, avg=3840.90, stdev=41.34,
samples=600
iops : min= 401, max= 1094, avg=957.38, stdev=10.37, samples=600
lat (usec) : 500=0.01%, 750=0.10%, 1000=0.70%
lat (msec) : 2=34.57%, 4=62.77%, 10=1.86%, 20=0.01%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,29069,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=3870KiB/s (3962kB/s), 3870KiB/s-3870KiB/s (3962kB/s-3962kB/s),
io=114MiB (119MB), run=30049-30049msec
with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share
--announce-submounts --inode-file-handles=prefer --syslog):
PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k
--directory="testdir" --ioengine=windowsaio --create_on_open=1
--file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of
threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 6 (f=318): [_(1),f(1),_(1),f(1),_(1),f(2),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8872: Fri May 2 15:55:39 2025
write: IOPS=964, BW=3859KiB/s (3952kB/s)(113MiB/30028msec); 0 zone resets
slat (nsec): min=0, max=1623.2k, avg=20705.88, stdev=16208.95
clat (usec): min=462, max=25441, avg=2312.40, stdev=778.69
lat (usec): min=489, max=25458, avg=2333.11, stdev=778.39
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1336], 10.00th=[ 1516], 20.00th=[ 1745],
| 30.00th=[ 1926], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2409],
| 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3163], 95.00th=[ 3523],
| 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5473], 99.95th=[ 7832],
| 99.99th=[25035]
bw ( KiB/s): min= 1180, max= 4385, per=98.89%, avg=3816.80, stdev=44.92,
samples=600
iops : min= 291, max= 1094, avg=952.10, stdev=11.25, samples=600
lat (usec) : 500=0.01%, 750=0.14%, 1000=0.86%
lat (msec) : 2=33.73%, 4=63.43%, 10=1.79%, 20=0.02%, 50=0.03%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,28969,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=3859KiB/s (3952kB/s), 3859KiB/s-3859KiB/s (3952kB/s-3952kB/s),
io=113MiB (119MB), run=30028-30028msec
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
