Plus from me too! I used the cpusets integration with PbsPro in my last job, and it was a godsend. This was on a large SMP machine, but same lessons apply to clusters. Applications get a defined set of CPUs - which they 'see' as being numbered from 0, and they get a defined amount of memory. If applications 'misbehave' then they get terminated when they run out of memory etc, and they can't stomp all over the cores being used by another application.
Also job tear-down seemed to be a lot better - ie when you collapse the cpuset then allthe processes associated with it, so you don't have to worry about orphan processes. ________________________________________ From: Christopher Samuel [[email protected]] Sent: 01 June 2016 00:53 To: slurm-dev Subject: [slurm-dev] Re: Increase size of running job/correcting incorrect resource allocations? On 31/05/16 18:42, Diego Zuccato wrote: > What I did is the other way around: I've used cpuset to "pin" the job to > the allocated CPUs. [...] > *Way* less need to watch closely the cluster :) +lots! We use Slurm's cgroup support to do just that, and Torque's cpuset support (which I agitated for) before that. Unless you're only ever allocating whole nodes to jobs then I'd strongly suggest taking that path (and use cgroups to restrict memory to what the job requested too). All the best, Chris -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP
