Hi, We would like to set process memory limits (vmemoryuse, in csh terms) on remote processes. Our batch system is torque/moab.
The nodes of our cluster each have 24GB of physical memory, of which 4GB is taken up by the kernel and the root file system. Note that these are diskless nodes, so no swap either. We can globally set the per-process limit to 2.5GB. This works fine if applications run "packed": 8 MPI tasks running on each 8-core node, for an aggregate limit of 20GB. However, if a job only wants to run 4 tasks, the soft limit can safely be raised to 5GB. 2 tasks, 10GB. 1 task, the full 20GB. Upping the soft limit in the batch script itself only affects the "head node" of the job. Since limits are not part of the "environment", I can find no way propagate them to remote nodes. If I understand how this all works, the remote processes are started by orted, and therefore inherit its limits. Is there any sort of orted configuration that can help here? Any other thoughts about how to approach this? Thanks! -- Best regards, David Turner User Services Group email: dptur...@lbl.gov NERSC Division phone: (510) 486-4027 Lawrence Berkeley Lab fax: (510) 486-4316
