NVidia seems to think so: https://wiki.lustre.org/images/3/30/LUG2021-Accelerating_AI_at_Scale-Bernauer-Kashinkunti.pdf
Cheers, Andreas On Jun 28, 2021, at 17:43, Vinayak.Kamath <[email protected]> wrote: Thanks for the prompt response, Andreas. Is Lustre a good choice for a non-volatile “cache” of ML training datasets? Thanks, Vinayak From: Andreas Dilger <[email protected]> Date: Monday, June 28, 2021 at 4:23 PM To: "Vinayak.Kamath" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: [EXTERNAL] Re: [lustre-discuss] What's your favorite distributed filesystem benchmark? On Jun 28, 2021, at 16:58, Vinayak.Kamath via lustre-discuss <[email protected]<mailto:[email protected]>> wrote: Greetings, Our team in the process of setting up an HPC system. We’re evaluating several distributed file systems(DFS), including Lustre. This is new territory for us and we’ve been learning a lot as we go. Those of you who have been breathing file system for years – are fio and ior good tools to benchmark a DFS? What are some common pitfalls to avoid while comparing filesystems? I suppose both these questions are complex topics worth writing papers and books on. Are there any good references that you can suggest? It depends on what your target application(s) are (no pun intended). Ideally, you would benchmark your actual application. For HPC, IOR is most commonly used for bandwidth, mdtest for file operations. These two are sometimes combined into an "aggregate" benchmark called IO500 that aggregates multiple different kinds of workloads (streaming, interleaved of large and small files, metadata create, lookup, unlink) (https://io500.org/, caveat I'm on the board for IO500 as well). Fio is mostly used for small block read/write IOPS measurements, and testing non-POSIX IO interfaces like libaio, io_uring, mmap, etc. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
