** Summary changed: - Poor write performance of storage volume with cephfs driver + Poor write performance of LXD storage volume with cephfs driver
** Description changed: The outlook: We deployed Microcloud solution for the client (LXD+Microceph+MicroOVN cluster) We are testing the perform of CephFS base The reproduction of this when having cluster established 1. Create LXD VM and attach cephfs volume sudo lxc launch --vm -c limits.cpu=8 -c limits.memory=8GiB ubuntu:24.04 test-fio-cephfs -s remote --network=OAM_OVN sudo lxc storage volume create remote-fs fio-cephfs-test size=1000GiB sudo lxc config device add test-fio-cephfs cephfs-data disk pool=remote-fs source=fio-cephfs-test path=/cephfs 2. then we install fio, git clone the repo repo with testing suite. cd /cephfs git clone https://github.com/canonical/ua-reviewkit cd ua-reviewkit/storage/fio and run ./run-tests.sh --name mytest --job all --class all --yes - The microclous consists of 3 nodes each: 1. Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz 2. 512GB of RAM 3. 2x960GB in hardware RAID1 for rootfs 4. 4x 5.8T SAS NVMe disks for microceph We tested the following scenarios 1. We mount the storage volume as above and run the tests on it While read speed was ok 991MB/s on 4K randread and 4400MB/s on 4M randread - the write speed was really poor + the write speed was really poor 1.15MB/s for 4k randrw (write) and 97.3MB/S for 4m randrw (write), the pure write was similar 2. We tried to improve by changing the logical sector to match physical (4K instead 512b) (cluster was redeployed) A small improvement almost everywhere up to 10%-15% 3. Using the 9p driver on VM instead of virtiofs. The LXD will use 9p driver if virtiofs driver is not available (so we blacklisted virtiofs on vm) The results are little better for 4k randrw (write) but for other worse 4. Direct mount cephfs filesystem (created by lxd: lxd_cephfs) on the baremetal host on one of the nodes Now the write are solid, but read performace is little worse that initially tested on VM 5. Direct mount cephfs filesystem on VM The results worse than on baremetal but still pretty good for write The results were summarized in this table: https://docs.google.com/spreadsheets/d/1AE_S9TJ2zSSOOo-Bu9Mig_55ciaqjl9CaQ_J27oNCPY/edit?gid=0#gid=0 + + + Extra info, + Ubuntu version Release: 24.04 + + Kernel: + 6.8.0-60-generic + + snaps: + $ snap list + Name Version Rev Tracking Publisher Notes + core22 20250425 1981 latest/stable canonical✓ base + core24 20250504 988 latest/stable canonical✓ base + lxd 5.21.3-c5ae129 33110 5.21/stable canonical✓ in-cohort + microceph 19.2.0+snap3b53da1c21 1355 squid/stable canonical✓ in-cohort + microcloud 2.1.0-3e8b183 1144 2/stable canonical✓ in-cohort + microovn 24.03.2+snapa2c59c105b 667 24.03/stable canonical✓ in-cohort + snapd 2.68.4 24505 latest/stable canonical✓ snapd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2112657 Title: Poor write performance of LXD storage volume with cephfs driver To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/2112657/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
