Running into an issue getting slurm power save features working on AWS.
Running slurm v17.02.2 on centos 7 and have it working just fine for my
initial partition of 10 t2micro nodes (t2micro is both the partition name
and the type of instance the nodes are on AWS which I use as the
default/debug queue).
My second partition is supposed to be elastic in that it will automatically
power on / create aws instances on demand. My resume/suspend scripts and
slurm.conf are available below but the problem I seem to be having is that
when a job is queued requesting a node in the cloud partition (which is
down/waiting on my resume script to come up) slurm does not allocate that
job to a node and never calls my resume script. Right now my resume script
is a test script that just logs that it was called, and I can test run the
script as the slurm user successfully. In addition to this if I run
$ scontrol update nodename="cloud-node-1" state=power_up
slurm actually DOES call my resume script on the specified node.
My best guess is something in my slurm.conf is conflicting so that when I
attempt to allocate a downed cloud node via srun a node is not allocated
(and my resume script does not get called). Any thoughts on this behavior?
Example input/output problem I am having: the following should run the
resume_program on a node in the m4xlarge partition. This entire partition
has the cloud state and none of the below trigger a slurm_resume event.
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
t2micro* up infinite 10 idle ip-10-0-1-[10-19]
m4xlarge up infinite 10 down ip-10-0-1-[20-29]
$ scontrol show node=ip-10-0-1-20
NodeName=ip-10-0-1-21 CoresPerSocket=4
CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=N/A
AvailableFeatures=cloud
ActiveFeatures=cloud
Gres=(null)
NodeAddr=ip-10-0-1-21 NodeHostName=ip-10-0-1-21 Port=0
RealMemory=15000 AllocMem=0 FreeMem=N/A Sockets=1 Boards=1
State=DOWN+POWER ThreadsPerCore=1 TmpDisk=0 Weight=8 Owner=N/A
MCS_label=N/A
Partitions=m4xlarge
BootTime=None SlurmdStartTime=None
CfgTRES=cpu=1,mem=15000M
AllocTRES=
CapWatts=n/a
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
Reason=cloud [root@2017-05-25T20:24:02]
$ srun -p m4xlarge -N 1 --test-only hostname
allocation failure: Requested node configuration is not available
$ srun -p m4xlarge -N 1 hostname
srun: Required node not available (down, drained or reserved)
srun: job 121 queued and waiting for resources
########### relevant? slurm.conf lines#############
## Elastic Compute
#exclude t2micro instances
SuspendExcNodes=ip-10-0-1-[10-19]
SuspendExcParts=t2micro,m4xlarge
#suspend resume locations
SuspendProgram=/usr/sbin/slurm_suspend
ResumeProgram=/usr/sbin/slurm_resume
ResumeRate=0
SuspendTime=200
ResumeTimeout=90
SuspendTimeout=90
TreeWidth=256
NodeName=ip-10-0-1-[10-19] CPUs=1 RealMemory=991 Sockets=1 \
CoresPerSocket=1 ThreadsPerCore=1 Feature=local State=UNKNOWN
NodeName=ip-10-0-1-[20-29] CPUs=1 RealMemory=15000 Sockets=1 \
CoresPerSocket=4 ThreadsPerCore=1 Weight=8 Feature=cloud State=CLOUD
PartitionName=t2micro Nodes=ip-10-0-1-[10-19] Default=YES MaxTime=INFINITE
State=UP
PartitionName=m4xlarge Nodes=ip-10-0-1-[20-29] Default=no MaxTime=INFINITE
############
#########/usr/sbin/slurm_resume
#!/bin/bash
echo "`date` Resume invoked $0 $*" >> /var/log/slurm/testing
hosts=`scontrol show hostnames $1`
for host in $hosts
do
echo "Resume $host" >> /var/log/slurm/testing
done
