Just enjoying the subject, I'm curious about what we can reach in disk performance when using NFS primary storage. We have two infrastructures with different purposes, one to instantiate normal VMs and other to be used by VMs involved with scientific researches running simulations, AI training, big databases loads etc.
I've made two simple tests here using the "fio" tool. The results are: SAS 7200RPM 2.7 TB x 12 -> PERC H710P Mini (Embedded) -> RAID6 -> NFS -> Primary Storage Pool (Normal server) Test inside one VM in this pool READ: bw=168MiB/s (177MB/s) WRITE: bw=186MiB/s (195MB/s) 4 TB SSD x 6 + SAS 10.000RPM 1.8 TB x 39 (HYBRID) -> RAID6 -> NFS -> Primary Storage Pool (Dell Unity XT 380 NAS) Test inside one VM in this pool READ: bw=417MiB/s WRITE: bw=461MiB/s Network connections among hosts and primary storage are 10 Gbps with jumbo frames. If anyone could share numbers like that I would appreciate. Can we reach better R/W performances? Ok, it looks obvious that with better hardware we can get better performance. But, in some tests made with servers using only NVMe units, I could see a limitation around 480 MB/s. This same server can reach around 4800 MB/s using the same fio test, locally. So, can we say the bottleneck is NFS? Thank you! Em ter., 2 de mai. de 2023 às 09:33, Pierre Le Fevre <[email protected]> escreveu: > Big thanks for all the suggestions :) > > We've ordered some NVME SSDs for write caching, this is something we missed > when setting up the machine originally. > Sounds like the setup should work other than that. > > Best, > Pierre > kthcloud > > On Mon, 1 May 2023 at 17:55, <[email protected]> wrote: > > > Me use flat network and jumbo frame too. > > El 29 de abr. de 2023 15:47 -0300, S.Fuller <[email protected]>, > > escribió: > > > Anything else different about the setup? Interface speeds? Routed vs > flat > > > network? MTU size being used by the network interfaces perhaps? > > > > > > - Steve > > > > > > On Fri, Apr 28, 2023 at 3:47 AM Pierre Le Fevre <[email protected]> > wrote: > > > > > > > Hi all, > > > > We're working on upgrading our storage solution to a proper network > > > > attached storage. > > > > Before this, we had an NFS share on some mounted disks on the > > management > > > > server. Our new setup is a NAS running TrueNAS (zfs) with 64 GB ram > and > > > > 8x8TB, 7200 rpm hard disks mounted to cloudstack over NFS. > > > > > > > > It seems the performance is however much lower than before, resulting > > in > > > > somewhat unusable virtual machines. In VMs, IOPS can be as low as <10 > > IOPS, > > > > vs 100 with the management server setup. > > > > > > > > Is there a recommended setup for primary storage to yield better > > > > performance? > > > > > > > > All the best > > > > Pierre, > > > > kthcloud > > > > > > > > > > > > > -- > > > Steve Fuller > > > [email protected] > > > -- __________________________ Aviso de confidencialidade Esta mensagem da Empresa Brasileira de Pesquisa Agropecuaria (Embrapa), empresa publica federal regida pelo disposto na Lei Federal no. 5.851, de 7 de dezembro de 1972, e enviada exclusivamente a seu destinatario e pode conter informacoes confidenciais, protegidas por sigilo profissional. Sua utilizacao desautorizada e ilegal e sujeita o infrator as penas da lei. Se voce a recebeu indevidamente, queira, por gentileza, reenvia-la ao emitente, esclarecendo o equivoco. Confidentiality note This message from Empresa Brasileira de Pesquisa Agropecuaria (Embrapa), a government company established under Brazilian law (5.851/72), is directed exclusively to its addressee and may contain confidential data, protected under professional secrecy rules. Its unauthorized use is illegal and may subject the transgressor to the law's penalties. If you are not the addressee, please send it back, elucidating the failure.
