JAson, note that compute-2018 is in IDLE* status - which means that it is not reachable. As Felip suggests, log into that compute node and tail -f /var/log/slurmd.log
I would also suggest on your master node running an scontrol to set that node as DRAIN then RESUME, then log into the node and (re)start the slurmd daemon I have had to do that when nodes have 'got their knickers in a twist' when I was playing around with the power up/down at idle settings in the past. On 8 May 2017 at 16:02, Felip Moll <[email protected]> wrote: > Do you have any kind of firewall in your network? > > I would suggest it is a problem with dates but since you tested munge -n > we could discard that. > Can you anyway do a pdsh -w compute-* date |dshbak -c ? > Can you show nodes slurmd log output? > > > > *--Felip Moll Marquès* > Computer Science Engineer > E-Mail - [email protected] > WebPage - http://lipix.ciutadella.es > > 2017-05-02 17:19 GMT+02:00 Jason Bacon <[email protected]>: > >> >> >> I have a perplexing error since reimaging a compute node. I've found a >> few similar issues on Google, none of which were resolved. Everything >> seems to be fine until I try to run a job on it. >> >> Any suggestions about where to look for clues would be appreciated. >> >> Thanks, >> >> Jason >> >> [[email protected] bacon]# scontrol show nodes compute-2-17 >> NodeName=compute-2-17 Arch=x86_64 CoresPerSocket=4 >> CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.01 Features=(null) >> Gres=(null) >> NodeAddr=compute-2-17 NodeHostName=compute-2-17 Version=15.08 >> OS=Linux RealMemory=24000 AllocMem=0 FreeMem=18357 Sockets=2 Boards=1 >> State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A >> BootTime=2017-04-22T18:55:28 SlurmdStartTime=2017-04-28T11:19:33 >> CapWatts=n/a >> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 >> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s >> >> >> [[email protected] bacon]# scontrol show nodes compute-2-18 >> NodeName=compute-2-18 Arch=x86_64 CoresPerSocket=4 >> CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.05 Features=(null) >> Gres=(null) >> NodeAddr=compute-2-18 NodeHostName=compute-2-18 Version=15.08 >> OS=Linux RealMemory=24000 AllocMem=0 FreeMem=23736 Sockets=2 Boards=1 >> State=IDLE* ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A >> BootTime=2017-04-28T11:14:23 SlurmdStartTime=2017-04-28T11:19:43 >> CapWatts=n/a >> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 >> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s >> >> >> [[email protected] bacon]# munge -n | ssh compute-2-17 unmunge >> STATUS: Success (0) >> ENCODE_HOST: login.avi.hpc.uwm.edu (129.89.58.18) >> ENCODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436) >> DECODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436) >> TTL: 300 >> CIPHER: aes128 (4) >> MAC: sha1 (3) >> ZIP: none (0) >> UID: root (0) >> GID: root (0) >> LENGTH: 0 >> >> [[email protected] bacon]# munge -n | ssh compute-2-18 unmunge >> STATUS: Success (0) >> ENCODE_HOST: login.avi.hpc.uwm.edu (129.89.58.18) >> ENCODE_TIME: 2017-04-28 11:20:39 -0500 (1493396439) >> DECODE_TIME: 2017-04-28 11:20:40 -0500 (1493396440) >> TTL: 300 >> CIPHER: aes128 (4) >> MAC: sha1 (3) >> ZIP: none (0) >> UID: root (0) >> GID: root (0) >> LENGTH: 0 >> >> [[email protected] bacon]# sinfo >> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST >> batch* up infinite 8 drng compute-5-[07-14] >> batch* up infinite 29 mix compute-2-[03-14],compute-5-[1 >> 5-28,31,33-34] >> batch* up infinite 66 alloc compute-1-[01-05],compute-2-[1 >> 9-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32] >> batch* up infinite 37 idle compute-1-[06-36],compute-2-[0 >> 1-02,15-18] >> batch-nice up infinite 8 drng compute-5-[07-14] >> batch-nice up infinite 29 mix compute-2-[03-14],compute-5-[1 >> 5-28,31,33-34] >> batch-nice up infinite 66 alloc compute-1-[01-05],compute-2-[1 >> 9-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32] >> batch-nice up infinite 37 idle compute-1-[06-36],compute-2-[0 >> 1-02,15-18] >> highmem up infinite 2 idle compute-5-[35-36] >> highmem-nice up infinite 2 idle compute-5-[35-36] >> >> [[email protected] bacon]# srun --nodelist=compute-2-17 hostname >> compute-2-17.avi >> [[email protected] bacon]# srun --nodelist=compute-2-18 hostname >> srun: error: Task launch for 360081.0 failed on node compute-2-18: >> Communication connection failure >> srun: error: Application launch failed: Communication connection failure >> srun: Job step aborted: Waiting up to 32 seconds for job step to finish. >> >> /var/log/slurm/slurmctld: >> >> [2017-04-28T11:25:09.002] job_step_signal step 360081.0 not found >> [2017-04-28T11:25:09.006] job_complete: JobID=360081 State=0x1 NodeCnt=1 >> WTERMSIG 105 >> [2017-04-28T11:25:09.054] job_complete: JobID=360081 State=0x8003 >> NodeCnt=1 done >> [2017-04-28T11:25:09.054] agent/is_node_resp: node:compute-2-18 >> RPC:REQUEST_TERMINATE_JOB : Communication connection failure >> [2017-04-28T11:25:42.153] error: Nodes compute-2-18 not responding >> >> -- >> Earth is a beta site. >> > >
