On a cluster I managed (without Lustre), we had many problems with users running nodes out of ram which often killed the node. We added cgroup support to slurm and those problems disappeared. Nearly 100% of the time get a cgroup OOM instead of a kernel OOM and the nodes would stay up and stable. This became doubly important when we started allowing jobs to share nodes and didn't want job A to be able to crash job B.
I've tried similar on a Lustre enabled cluster and it seems like the memory used by Lustre (which I believe is in the kernel and outside of the job's cgroup). I think part of the problem is I believe Lustre caches metadata in the linux page cache, but not data. I've tried reducing the ram available to slurm, but still getting kernel OOMs instead of cgroup OOMs.
Anyone have a suggestion for fixing this? Is there any way to limit Lustre's memory use in the kernel? Or force that caching into userspace and inside the cgroup? Or possibly out of ram and onto a client local NVMe?
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
