Hi all I'm using a small slurm cluster that spins up extra spot instance nodes on EC2 using the elastic computing <http://schedmd.com/slurmdocs/elastic_computing.html> feature. Most things seem to be working well, but when I start slurmctld, it powers up *all* of the nodes, launching the maximum possible number of instances at once (and I'm charged for an hour of use for each of them).
Is there a way of starting slurmctld that starts without running the slurm_resume script over all of the nodes? Using -c flag seems to do the trick, but that seems heavy-handed. Will starting slurmctld with '-c' wipe away things like queued jobs? A subset of slurm.conf: ... SuspendProgram=/usr/sbin/slurm_suspend ResumeProgram=/usr/sbin/slurm_resume ResumeTimeout=1200 SuspendTime=600 NodeName=ip-10-0-0-[5-7] Weight=2 CPUs=1 RealMemory=3764 Feature=cloud State=CLOUD PartitionName=basic Nodes=ip-10-0-0-[5-7] Default=YES MaxTime=INFINITE ... >From the slurmctld log (by own highlighting): ... slurmctld: Munge cryptographic signature plugin loaded slurmctld: preempt/none loaded slurmctld: Checkpoint plugin loaded: checkpoint/none slurmctld: Job accounting gather NOT_INVOKED plugin loaded slurmctld: ExtSensors NONE plugin loaded slurmctld: debug: No backup controller to shutdown slurmctld: switch NONE plugin loaded slurmctld: debug: Reading slurm.conf file: /etc/slurm-llnl/slurm.conf slurmctld: Recovered state of 3 nodes slurmctld: Recovered information about 0 jobs slurmctld: debug: Updating partition uid access list slurmctld: Recovered state of 0 reservations slurmctld: State of 0 triggers recovered slurmctld: read_slurm_conf: backup_controller not specified. slurmctld: Running as primary controller slurmctld: debug: Priority BASIC plugin loaded slurmctld: debug2: slurmctld listening on 0.0.0.0:6817 slurmctld: Power save mode: 3 nodes *slurmctld: power_save: waking nodes ip-10-0-0-[5-7]* slurmctld: debug: Spawning registration agent for ip-10-0-0-[5-7] 3 hosts ... Any help is appreciated. -r
