Thanks to those who replied to my question, I've figured out the problem. Slurmctld was getting confused when the ip address of a node changed, this could happen if, for example, node15 was running in us-east1-b with an ip address of 10.142.0.3, and after being deleted and relaunched in us-central1-a node15 it would have an ip address 10.128.0.2. Slurm would continue to treat node15 as if it had the old ip address.
I finally found this because the ip address of two nodes managed to get switched while I was torture testing my virtual machine launching code by deleting multiple nodes at one time to simulate preemption. After the nodes were relaunched the log starting containing messages such as "error: Batch completion for job 47137 sent from wrong node (node16 rather than node15). Was the job requeued due to node failure?". After I modified my code to never delete a preempted node but Instead it leave it in the stopped stage and launch a virtual machine with a new, unique node name I've not had any problems. BTW, for anyone interested in how I configured slurm this is what I have done: In slurm.conf I declare the nodes as normal: NodeName=node[1-30] CPUs=16 RealMemory=94000 State=UNKNOWN PartitionName=default Nodes=node[1-30] Default=YES MaxTime=INFINITE State=UP The number of nodes should be the number of nodes you want to run in your cluster plus some spares (I've found that 50% spares is enough, though if you are getting frequent preemptions you'll want to make it larger). After changing this value you'll need to restart the slurmctld (I'm assuming there are no compute nodes currently running, if there are I believe you'll need to run scontrol reconfigure). Making the number of nodes value excessively large will cause the slurmctld restart to take a long time, since it tries to determine the ip address for each node when it starts. Since the nodes don't yet exist this will fail, which doesn't seem to bother slurmctld, but each of these checks takes about 2 seconds. I then launch the nodes that I'll need for the run and after each node is launched and is responding to pings I inform slurmctld that it's available: sudo scontrol update nodename=node1 state=down reason=new sleep 5 sudo scontrol update nodename=node1 state=resume When a node is preempted (which I detect by it no long responding to pings) I set it to drain: sudo scontrol update nodename=node1 state=drain reason=ping I set any nodes that have drained to state=fail reason=sleep. When a node is preempted I look through the list of nodes that are state=fail reason=sleep and if there is one in a region/zone that hasn't had a recent preemption I start that node and after it responds to pings perform the same state=down; sleep 5; state=resume steps as for a new node. If there are no nodes available, I launch a new one, hence the need for extra nodes in the slurm.conf file I believe it's important to launch the nodes in numerical order; I've had problems getting slurmctkd to recognize a newly launched node otherwise. If anyone has any questions or suggestions please email me or reply to this post. marcos On Tue, May 2, 2017 at 2:27 PM, Marcos Woehrmann <[email protected]> wrote: > I've setup a slurm cluster on Google Cloud. Everything is working great > if I use persistent instances. > > However, when I use preemptible instances as compute nodes I run into > issues when an instance is preempted. Jobs running on that instance > sometimes get stuck in the RUNNING state forever. If I scancel the job its > state changes to COMPLETING but it's then stuck in that state. > > Here's the scontrol output for one of stuck jobs: > > mwoehr@master:~$ scontrol show jobid --d 37412 > JobId=37412 JobName=job.sh > UserId=mwoehr(1001) GroupId=mwoehr(1002) MCS_label=N/A > Priority=4294901753 Nice=0 Account=(null) QOS=(null) > JobState=RUNNING Reason=None Dependency=(null) > Requeue=1 Restarts=2 BatchFlag=2 Reboot=0 ExitCode=0:0 > DerivedExitCode=0:0 > RunTime=11:00:21 TimeLimit=UNLIMITED TimeMin=N/A > SubmitTime=2017-05-02T05:18:58 EligibleTime=2017-05-02T05:20:59 > StartTime=2017-05-02T05:49:13 EndTime=Unknown Deadline=N/A > PreemptTime=None SuspendTime=None SecsPreSuspend=0 > Partition=debug AllocNode:Sid=master:24531 > ReqNodeList=(null) ExcNodeList=(null) > NodeList=node4 > BatchHost=node4 > NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > TRES=cpu=1,mem=4000M,node=1 > Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* > Nodes=node4 CPU_IDs=1 Mem=4000 > MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 > Features=(null) Gres=(null) Reservation=(null) > OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) > Command=/home/mwoehr/job.sh > WorkDir=/home/mwoehr > StdErr=/home/mwoehr/slurm-37412.out > StdIn=/dev/null > StdOut=/home/mwoehr/slurm-37412.out > Power= > > > And applicable node: > > mwoehr@master:~$ sinfo -N -l -n node4 > Tue May 2 16:54:33 2017 > NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK > WEIGHT AVAIL_FE REASON > node4 1 debug* completing* 8 8:1:1 28000 0 > 1 (null) NO NETWORK ADDRESS F > > > The slurmctld log file doesn't seem very helpful: > > May 2 16:32:29 master slurmctld: error: Nodes node4 not responding > May 2 16:32:32 master slurmctld: error: Unable to resolve "node4": > Unknown server error > May 2 16:32:32 master slurmctld: error: fwd_tree_thread: can't find > address for host node4, check slurm.conf > > > > Does anyone have any suggestions on how to debug this issue? > > > marcos > > -- > Marcos H. Woehrmann [email protected] > > "It is written entirely in the present tense in strange, overwrought > language that seems always to be tiptoeing around on the brink of > incoherence." -- Bill Bryson, At Home > -- Marcos H. Woehrmann [email protected] "It is written entirely in the present tense in strange, overwrought language that seems always to be tiptoeing around on the brink of incoherence." -- Bill Bryson, At Home
