So, I have not worked with GCE, but sounds like what's happening is that the instance that "node4" was running on was de-provisioned, _including_ the host entry for "node4".
Slurm *really* doesn't like it when the hostname doesn't resolve. It can't even mark the node as down. Heck, won't even start when it can't resolve node names. To clear it... here[1] is an old thread suggesting you use `scontrol down node4` to tell slurmctld to ignore job status updates. Might be something better in more recent versions of Slurm (what version are you using?). But again, with an un-resolvable hostname... YMMV. Also: are you using "state=cloud" in your node definitions? I'm not sure that helps when the host name disappears. Curious to see how this works out for you- we're dabbling in Slurm/EC2 and I know that eventually I'll get asked to put things in the spot market... I believe there is some event that EC2 emits when an instance is set to be killed- GCE may have a similar capability. Might give you the time necessary to clear off jobs or down the node before GCE kills it. Michael [1] https://groups.google.com/forum/#!topic/slurm-devel/PE1E3sLHN9k On Tue, May 2, 2017 at 2:27 PM, Marcos Woehrmann <[email protected]> wrote: > I've setup a slurm cluster on Google Cloud. Everything is working great > if I use persistent instances. > > However, when I use preemptible instances as compute nodes I run into > issues when an instance is preempted. Jobs running on that instance > sometimes get stuck in the RUNNING state forever. If I scancel the job its > state changes to COMPLETING but it's then stuck in that state. > > Here's the scontrol output for one of stuck jobs: > > mwoehr@master:~$ scontrol show jobid --d 37412 > JobId=37412 JobName=job.sh > UserId=mwoehr(1001) GroupId=mwoehr(1002) MCS_label=N/A > Priority=4294901753 Nice=0 Account=(null) QOS=(null) > JobState=RUNNING Reason=None Dependency=(null) > Requeue=1 Restarts=2 BatchFlag=2 Reboot=0 ExitCode=0:0 > DerivedExitCode=0:0 > RunTime=11:00:21 TimeLimit=UNLIMITED TimeMin=N/A > SubmitTime=2017-05-02T05:18:58 EligibleTime=2017-05-02T05:20:59 > StartTime=2017-05-02T05:49:13 EndTime=Unknown Deadline=N/A > PreemptTime=None SuspendTime=None SecsPreSuspend=0 > Partition=debug AllocNode:Sid=master:24531 > ReqNodeList=(null) ExcNodeList=(null) > NodeList=node4 > BatchHost=node4 > NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > TRES=cpu=1,mem=4000M,node=1 > Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* > Nodes=node4 CPU_IDs=1 Mem=4000 > MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 > Features=(null) Gres=(null) Reservation=(null) > OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) > Command=/home/mwoehr/job.sh > WorkDir=/home/mwoehr > StdErr=/home/mwoehr/slurm-37412.out > StdIn=/dev/null > StdOut=/home/mwoehr/slurm-37412.out > Power= > > > And applicable node: > > mwoehr@master:~$ sinfo -N -l -n node4 > Tue May 2 16:54:33 2017 > NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK > WEIGHT AVAIL_FE REASON > node4 1 debug* completing* 8 8:1:1 28000 0 > 1 (null) NO NETWORK ADDRESS F > > > The slurmctld log file doesn't seem very helpful: > > May 2 16:32:29 master slurmctld: error: Nodes node4 not responding > May 2 16:32:32 master slurmctld: error: Unable to resolve "node4": > Unknown server error > May 2 16:32:32 master slurmctld: error: fwd_tree_thread: can't find > address for host node4, check slurm.conf > > > > Does anyone have any suggestions on how to debug this issue? > > > marcos > > -- > Marcos H. Woehrmann [email protected] > > "It is written entirely in the present tense in strange, overwrought > language that seems always to be tiptoeing around on the brink of > incoherence." -- Bill Bryson, At Home >
