Michael Gutteridge wrote on May, 02 17:07:38: > So, I have not worked with GCE, but sounds like what's happening is that > the instance that "node4" was running on was de-provisioned, _including_ > the host entry for "node4". > > Slurm *really* doesn't like it when the hostname doesn't resolve. It can't > even mark the node as down. Heck, won't even start when it can't resolve > node names. > > To clear it... here[1] is an old thread suggesting you use `scontrol down > node4` to tell slurmctld to ignore job status updates. Might be something > better in more recent versions of Slurm (what version are you using?). But > again, with an un-resolvable hostname... YMMV. > > Also: are you using "state=cloud" in your node definitions? I'm not sure > that helps when the host name disappears. > > Curious to see how this works out for you- we're dabbling in Slurm/EC2 and > I know that eventually I'll get asked to put things in the spot market... > I believe there is some event that EC2 emits when an instance is set to be > killed- GCE may have a similar capability. Might give you the time > necessary to clear off jobs or down the node before GCE kills it.
You're looking for shutdown scripts for your instances. https://cloud.google.com/compute/docs/instances/create-start-preemptible-instance#handle_preemption > > Michael > > > [1] https://groups.google.com/forum/#!topic/slurm-devel/PE1E3sLHN9k > > > On Tue, May 2, 2017 at 2:27 PM, Marcos Woehrmann <[email protected]> wrote: > > > I've setup a slurm cluster on Google Cloud. Everything is working great > > if I use persistent instances. > > > > However, when I use preemptible instances as compute nodes I run into > > issues when an instance is preempted. Jobs running on that instance > > sometimes get stuck in the RUNNING state forever. If I scancel the job its > > state changes to COMPLETING but it's then stuck in that state. > > > > Here's the scontrol output for one of stuck jobs: > > > > mwoehr@master:~$ scontrol show jobid --d 37412 > > JobId=37412 JobName=job.sh > > UserId=mwoehr(1001) GroupId=mwoehr(1002) MCS_label=N/A > > Priority=4294901753 Nice=0 Account=(null) QOS=(null) > > JobState=RUNNING Reason=None Dependency=(null) > > Requeue=1 Restarts=2 BatchFlag=2 Reboot=0 ExitCode=0:0 > > DerivedExitCode=0:0 > > RunTime=11:00:21 TimeLimit=UNLIMITED TimeMin=N/A > > SubmitTime=2017-05-02T05:18:58 EligibleTime=2017-05-02T05:20:59 > > StartTime=2017-05-02T05:49:13 EndTime=Unknown Deadline=N/A > > PreemptTime=None SuspendTime=None SecsPreSuspend=0 > > Partition=debug AllocNode:Sid=master:24531 > > ReqNodeList=(null) ExcNodeList=(null) > > NodeList=node4 > > BatchHost=node4 > > NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > > TRES=cpu=1,mem=4000M,node=1 > > Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* > > Nodes=node4 CPU_IDs=1 Mem=4000 > > MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 > > Features=(null) Gres=(null) Reservation=(null) > > OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) > > Command=/home/mwoehr/job.sh > > WorkDir=/home/mwoehr > > StdErr=/home/mwoehr/slurm-37412.out > > StdIn=/dev/null > > StdOut=/home/mwoehr/slurm-37412.out > > Power= > > > > > > And applicable node: > > > > mwoehr@master:~$ sinfo -N -l -n node4 > > Tue May 2 16:54:33 2017 > > NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK > > WEIGHT AVAIL_FE REASON > > node4 1 debug* completing* 8 8:1:1 28000 0 > > 1 (null) NO NETWORK ADDRESS F > > > > > > The slurmctld log file doesn't seem very helpful: > > > > May 2 16:32:29 master slurmctld: error: Nodes node4 not responding > > May 2 16:32:32 master slurmctld: error: Unable to resolve "node4": > > Unknown server error > > May 2 16:32:32 master slurmctld: error: fwd_tree_thread: can't find > > address for host node4, check slurm.conf > > > > > > > > Does anyone have any suggestions on how to debug this issue? > > > > > > marcos > > > > -- > > Marcos H. Woehrmann [email protected] > > > > "It is written entirely in the present tense in strange, overwrought > > language that seems always to be tiptoeing around on the brink of > > incoherence." -- Bill Bryson, At Home > > -- Chris Scheller Application Programmer/Analyst Senior Center for Statistical Genetics Department of Biostatistics School of Public Health University of Michigan Phone: (734) 615-7439 Office: M4218
