Hello, I am experimenting with slurm's elastic computing capabilities in GCE and having some problem with auto-scaling. I just kicked off a job that is sitting and waiting for resources. I'm unclear on why it's not running or spinning up new cloud nodes. All of the nodes configured in slurm.conf are set up as cloud nodes. I've confirmed that the resume and suspend scripts work before and I'm not sure why sometimes slurm kicks off jobs and sometimes it doesn't.
### sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST prod* up infinite 4 idle~ c-n-[000-003] test up infinite 4 idle~ c-n-[000-003] ### squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 93 prod kent-abc kstevens PD 0:00 3 (Resources) 94 prod kent-abc kstevens PD 0:00 3 (Resources) 95 prod kent-abc kstevens PD 0:00 3 (Resources) ### slurm.conf # POWER SAVE SUPPORT FOR IDLE NODES (optional) SuspendProgram=/opt/bin/suspend.sh ResumeProgram=/opt/bin/resume.sh SuspendTime=60 SuspendTimeout=360 ResumeTimeout=360 ResumeRate=10 #SuspendExcNodes= #SuspendExcParts= #SuspendRate= TreeWidth=128 NodeName=c-n-0[00-14] CPUs=16 Feature=cloud State=CLOUD PartitionName=prod Nodes=c-n-0[00-14] Default=YES MaxTime=INFINITE PRIORITY=5 PartitionName=test Nodes=c-n-0[00-14] Default=no MaxTime=INFINITE PRIORITY=5 ### scontrol show job 9 JobId=93 JobName=kent-abc UserId=kstevens(500) GroupId=kstevens(500) Priority=1000 Nice=0 Account=(null) QOS=(null) JobState=PENDING Reason=Resources Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A SubmitTime=2016-04-12T16:44:33 EligibleTime=2016-04-12T16:44:33 StartTime=2016-04-13T17:22:56 EndTime=Unknown PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=prod AllocNode:Sid=c-ws-00:27312 ReqNodeList=(null) ExcNodeList=(null) NodeList=(null) NumNodes=3-3 NumCPUs=3 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=(null) Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0 Features=(null) Gres=(null) Reservation=(null) Shared=0 Contiguous=0 Licenses=(null) Network=(null) Nothing in my logs shows any attempts to run the resume command to spin up nodes for job 93. thanks, Houston Dee
