Public bug reported: The outlook:
We deployed Microcloud solution for the client (LXD+Microceph+MicroOVN cluster) We are testing the perform of CephFS base The reproduction of this when having cluster established 1. Create LXD VM and attach cephfs volume sudo lxc launch --vm -c limits.cpu=8 -c limits.memory=8GiB ubuntu:24.04 test-fio-cephfs -s remote --network=OAM_OVN sudo lxc storage volume create remote-fs fio-cephfs-test size=1000GiB sudo lxc config device add test-fio-cephfs cephfs-data disk pool=remote-fs source=fio-cephfs-test path=/cephfs 2. then we install fio, git clone the repo repo with testing suite. cd /cephfs git clone https://github.com/canonical/ua-reviewkit cd ua-reviewkit/storage/fio and run ./run-tests.sh --name mytest --job all --class all --yes The microclous consists of 3 nodes each: 1. Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz 2. 512GB of RAM 3. 2x960GB in hardware RAID1 for rootfs 4. 4x 5.8T SAS NVMe disks for microceph We tested the following scenarios 1. We mount the storage volume as above and run the tests on it While read speed was ok 991MB/s on 4K randread and 4400MB/s on 4M randread the write speed was really poor 1.15MB/s for 4k randrw (write) and 97.3MB/S for 4m randrw (write), the pure write was similar 2. We tried to improve by changing the logical sector to match physical (4K instead 512b) (cluster was redeployed) A small improvement almost everywhere up to 10%-15% 3. Using the 9p driver on VM instead of virtiofs. The LXD will use 9p driver if virtiofs driver is not available (so we blacklisted virtiofs on vm) The results are little better for 4k randrw (write) but for other worse 4. Direct mount cephfs filesystem (created by lxd: lxd_cephfs) on the baremetal host on one of the nodes Now the write are solid, but read performace is little worse that initially tested on VM 5. Direct mount cephfs filesystem on VM The results worse than on baremetal but still pretty good for write The results were summarized in this table: https://docs.google.com/spreadsheets/d/1AE_S9TJ2zSSOOo-Bu9Mig_55ciaqjl9CaQ_J27oNCPY/edit?gid=0#gid=0 ** Affects: lxd (Ubuntu) Importance: Undecided Status: New ** Summary changed: - Poor performance of storage volume with cephfs driver + Poor write performance of storage volume with cephfs driver -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2112657 Title: Poor write performance of storage volume with cephfs driver To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/2112657/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
