I've setup a slurm cluster on Google Cloud. Everything is working great if
I use persistent instances.
However, when I use preemptible instances as compute nodes I run into
issues when an instance is preempted. Jobs running on that instance
sometimes get stuck in the RUNNING state forever. If I scancel the job its
state changes to COMPLETING but it's then stuck in that state.
Here's the scontrol output for one of stuck jobs:
mwoehr@master:~$ scontrol show jobid --d 37412
JobId=37412 JobName=job.sh
UserId=mwoehr(1001) GroupId=mwoehr(1002) MCS_label=N/A
Priority=4294901753 Nice=0 Account=(null) QOS=(null)
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=2 BatchFlag=2 Reboot=0 ExitCode=0:0
DerivedExitCode=0:0
RunTime=11:00:21 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2017-05-02T05:18:58 EligibleTime=2017-05-02T05:20:59
StartTime=2017-05-02T05:49:13 EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=debug AllocNode:Sid=master:24531
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node4
BatchHost=node4
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=1,mem=4000M,node=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
Nodes=node4 CPU_IDs=1 Mem=4000
MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/mwoehr/job.sh
WorkDir=/home/mwoehr
StdErr=/home/mwoehr/slurm-37412.out
StdIn=/dev/null
StdOut=/home/mwoehr/slurm-37412.out
Power=
And applicable node:
mwoehr@master:~$ sinfo -N -l -n node4
Tue May 2 16:54:33 2017
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT
AVAIL_FE REASON
node4 1 debug* completing* 8 8:1:1 28000 0 1
(null) NO NETWORK ADDRESS F
The slurmctld log file doesn't seem very helpful:
May 2 16:32:29 master slurmctld: error: Nodes node4 not responding
May 2 16:32:32 master slurmctld: error: Unable to resolve "node4": Unknown
server error
May 2 16:32:32 master slurmctld: error: fwd_tree_thread: can't find
address for host node4, check slurm.conf
Does anyone have any suggestions on how to debug this issue?
marcos
--
Marcos H. Woehrmann [email protected]
"It is written entirely in the present tense in strange, overwrought
language that seems always to be tiptoeing around on the brink of
incoherence." -- Bill Bryson, At Home
