Do you have any kind of firewall in your network? I would suggest it is a problem with dates but since you tested munge -n we could discard that. Can you anyway do a pdsh -w compute-* date |dshbak -c ? Can you show nodes slurmd log output?
*--Felip Moll Marquès* Computer Science Engineer E-Mail - [email protected] WebPage - http://lipix.ciutadella.es 2017-05-02 17:19 GMT+02:00 Jason Bacon <[email protected]>: > > > I have a perplexing error since reimaging a compute node. I've found a > few similar issues on Google, none of which were resolved. Everything > seems to be fine until I try to run a job on it. > > Any suggestions about where to look for clues would be appreciated. > > Thanks, > > Jason > > [[email protected] bacon]# scontrol show nodes compute-2-17 > NodeName=compute-2-17 Arch=x86_64 CoresPerSocket=4 > CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.01 Features=(null) > Gres=(null) > NodeAddr=compute-2-17 NodeHostName=compute-2-17 Version=15.08 > OS=Linux RealMemory=24000 AllocMem=0 FreeMem=18357 Sockets=2 Boards=1 > State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > BootTime=2017-04-22T18:55:28 SlurmdStartTime=2017-04-28T11:19:33 > CapWatts=n/a > CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s > > > [[email protected] bacon]# scontrol show nodes compute-2-18 > NodeName=compute-2-18 Arch=x86_64 CoresPerSocket=4 > CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.05 Features=(null) > Gres=(null) > NodeAddr=compute-2-18 NodeHostName=compute-2-18 Version=15.08 > OS=Linux RealMemory=24000 AllocMem=0 FreeMem=23736 Sockets=2 Boards=1 > State=IDLE* ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A > BootTime=2017-04-28T11:14:23 SlurmdStartTime=2017-04-28T11:19:43 > CapWatts=n/a > CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 > ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s > > > [[email protected] bacon]# munge -n | ssh compute-2-17 unmunge > STATUS: Success (0) > ENCODE_HOST: login.avi.hpc.uwm.edu (129.89.58.18) > ENCODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436) > DECODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436) > TTL: 300 > CIPHER: aes128 (4) > MAC: sha1 (3) > ZIP: none (0) > UID: root (0) > GID: root (0) > LENGTH: 0 > > [[email protected] bacon]# munge -n | ssh compute-2-18 unmunge > STATUS: Success (0) > ENCODE_HOST: login.avi.hpc.uwm.edu (129.89.58.18) > ENCODE_TIME: 2017-04-28 11:20:39 -0500 (1493396439) > DECODE_TIME: 2017-04-28 11:20:40 -0500 (1493396440) > TTL: 300 > CIPHER: aes128 (4) > MAC: sha1 (3) > ZIP: none (0) > UID: root (0) > GID: root (0) > LENGTH: 0 > > [[email protected] bacon]# sinfo > PARTITION AVAIL TIMELIMIT NODES STATE NODELIST > batch* up infinite 8 drng compute-5-[07-14] > batch* up infinite 29 mix compute-2-[03-14],compute-5-[1 > 5-28,31,33-34] > batch* up infinite 66 alloc compute-1-[01-05],compute-2-[1 > 9-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32] > batch* up infinite 37 idle compute-1-[06-36],compute-2-[0 > 1-02,15-18] > batch-nice up infinite 8 drng compute-5-[07-14] > batch-nice up infinite 29 mix compute-2-[03-14],compute-5-[1 > 5-28,31,33-34] > batch-nice up infinite 66 alloc compute-1-[01-05],compute-2-[1 > 9-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32] > batch-nice up infinite 37 idle compute-1-[06-36],compute-2-[0 > 1-02,15-18] > highmem up infinite 2 idle compute-5-[35-36] > highmem-nice up infinite 2 idle compute-5-[35-36] > > [[email protected] bacon]# srun --nodelist=compute-2-17 hostname > compute-2-17.avi > [[email protected] bacon]# srun --nodelist=compute-2-18 hostname > srun: error: Task launch for 360081.0 failed on node compute-2-18: > Communication connection failure > srun: error: Application launch failed: Communication connection failure > srun: Job step aborted: Waiting up to 32 seconds for job step to finish. > > /var/log/slurm/slurmctld: > > [2017-04-28T11:25:09.002] job_step_signal step 360081.0 not found > [2017-04-28T11:25:09.006] job_complete: JobID=360081 State=0x1 NodeCnt=1 > WTERMSIG 105 > [2017-04-28T11:25:09.054] job_complete: JobID=360081 State=0x8003 > NodeCnt=1 done > [2017-04-28T11:25:09.054] agent/is_node_resp: node:compute-2-18 > RPC:REQUEST_TERMINATE_JOB : Communication connection failure > [2017-04-28T11:25:42.153] error: Nodes compute-2-18 not responding > > -- > Earth is a beta site. >
