Hi Antonia Hmm....we set the Default Memory Per CPU for the partitions as well as having the node memory populated, so I may be unable to help here, but I'm curious as to why your node reports "RealMemory=1" - but I don't know if that is 'normal' in some case, to be honest. If I request an impossible memory configuration I get the message you do at job submission. Wonder if passing the '-v -v -v' to sbatch provides useful debugging output on the what Slurm assesses the job requirements to be and whether specifying compliant memory requirements for the job helps.
Sorry I can't be more help. ~~ Ade ________________________________ From: Antonia Mey <[email protected]> Sent: 18 August 2016 16:30:14 To: slurm-dev Subject: [slurm-dev] Re: jobs rejected for no reason Hi Ade, looking at the node configuration this should be ok: NodeName=node011 Arch=x86_64 CoresPerSocket=1 CPUAlloc=4 CPUErr=0 CPUTot=32 CPULoad=2.79 Features=(null) Gres=gpu:4 NodeAddr=node011 NodeHostName=node011 Version=14.11 OS=Linux RealMemory=1 AllocMem=0 Sockets=32 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 BootTime=2015-10-30T09:30:14 SlurmdStartTime=2015-10-30T12:02:36 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s On 18 Aug 2016, at 16:18, Ade Fewings <[email protected]<mailto:[email protected]>> wrote: Hi Antonia I think it's quite likely to be something to do with the nodes in that partition, does 'scontrol show node=<nodeid>' show all the requested capabilities? ~~ Ade ________________________________ From: Antonia Mey <[email protected]<mailto:[email protected]>> Sent: 18 August 2016 15:55:12 To: slurm-dev Subject: [slurm-dev] jobs rejected for no reason Hi all, I am a bit out of my depth here and apologies if this is a very trivial problem. Slurm rejects jobs due to insufficient resources (sbatch: error: Batch job submission failed: Requested node configuration is not available), when the partition should definitely accept the following job #!/bin/bash --login #SBATCH -n 8 #SBATCH -N 1 #SBATCH -o d66a.out #SBATCH -e d66a.err #SBATCH -p GTX #SBATCH --gres=gpu:1 #SBATCH --time 8:00:00 # Switch to current working directory scontrol show partition GTX gives: PartitionName=GTX AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL AllocNodes=ALL Default=NO DefaultTime=NONE DisableRootJobs=NO GraceTime=0 Hidden=NO MaxNodes=UNLIMITED MaxTime=2-00:00:00 MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED Nodes=node0[10-13] Priority=1 RootOnly=NO ReqResv=NO Shared=NO PreemptMode=OFF State=UP TotalCPUs=128 TotalNodes=4 SelectTypeParameters=N/A DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED I have a similar partition configured in the same way, where the job is happily running. Am I missing something essential in the way the partition is configured? Any thoughts? And thanks for any help. Best, Antonia -- Dr Antonia Mey University of Edinburgh Department of Chemistry Joseph Black Building Edinburgh EH9 3FJ Tel: +44 1316507748 Email: [email protected]<mailto:[email protected]> ________________________________ [HPC Wales - www.hpcwales.co.uk] <http://www.hpcwales.co.uk/> ________________________________ The contents of this email and any files transmitted with it are confidential and intended solely for the named addressee only. Unless you are the named addressee (or authorised to receive this on their behalf) you may not copy it or use it, or disclose it to anyone else. If you have received this email in error, please notify the sender by email or telephone. All emails sent by High Performance Computing Wales have been checked using an Anti-Virus system. We would advise you to run your own virus check before opening any attachments received as we will not in any event accept any liability whatsoever, once an email and/or attachment is received. High Performance Computing Wales is a private limited company incorporated in Wales on 8 March 2010 as company number 07181701. Our registered office is at Finance Office, Bangor University, Cae Derwen, College Road, Bangor, Gwynedd. LL57 2DG. UK. High Performance Computing Wales is part funded by the European Regional Development Fund through the Welsh Government. -- Dr Antonia Mey University of Edinburgh Department of Chemistry Joseph Black Building Edinburgh EH9 3FJ Tel: +44 1316507748 Email: [email protected]<mailto:[email protected]> ________________________________ [HPC Wales - www.hpcwales.co.uk] <http://www.hpcwales.co.uk> ________________________________ The contents of this email and any files transmitted with it are confidential and intended solely for the named addressee only. Unless you are the named addressee (or authorised to receive this on their behalf) you may not copy it or use it, or disclose it to anyone else. If you have received this email in error, please notify the sender by email or telephone. All emails sent by High Performance Computing Wales have been checked using an Anti-Virus system. We would advise you to run your own virus check before opening any attachments received as we will not in any event accept any liability whatsoever, once an email and/or attachment is received. High Performance Computing Wales is a private limited company incorporated in Wales on 8 March 2010 as company number 07181701. Our registered office is at Finance Office, Bangor University, Cae Derwen, College Road, Bangor, Gwynedd. LL57 2DG. UK. High Performance Computing Wales is part funded by the European Regional Development Fund through the Welsh Government.
