Following a tip from Moe, I actually traced the problem to an iptables
issue specific to the TCP port used by SLURM. It's resolved now.
Thanks for the suggestions, though.
On 05/08/17 09:01, Felip Moll wrote:
Re: [slurm-dev] Communication error
Do you have any kind of firewall in your network?
I would suggest it is a problem with dates but since you tested munge
-n we could discard that.
Can you anyway do a pdsh -w compute-* date |dshbak -c ?
Can you show nodes slurmd log output?
*
--
Felip Moll Marquès*
Computer Science Engineer
E-Mail - [email protected] <mailto:[email protected]>
WebPage - http://lipix.ciutadella.es
2017-05-02 17:19 GMT+02:00 Jason Bacon <[email protected]
<mailto:[email protected]>>:
I have a perplexing error since reimaging a compute node. I've
found a few similar issues on Google, none of which were
resolved. Everything seems to be fine until I try to run a job on it.
Any suggestions about where to look for clues would be appreciated.
Thanks,
Jason
[[email protected] bacon]# scontrol show nodes compute-2-17
NodeName=compute-2-17 Arch=x86_64 CoresPerSocket=4
CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.01 Features=(null)
Gres=(null)
NodeAddr=compute-2-17 NodeHostName=compute-2-17 Version=15.08
OS=Linux RealMemory=24000 AllocMem=0 FreeMem=18357 Sockets=2
Boards=1
State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
BootTime=2017-04-22T18:55:28 SlurmdStartTime=2017-04-28T11:19:33
CapWatts=n/a
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
[[email protected] bacon]# scontrol show nodes compute-2-18
NodeName=compute-2-18 Arch=x86_64 CoresPerSocket=4
CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.05 Features=(null)
Gres=(null)
NodeAddr=compute-2-18 NodeHostName=compute-2-18 Version=15.08
OS=Linux RealMemory=24000 AllocMem=0 FreeMem=23736 Sockets=2
Boards=1
State=IDLE* ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A
BootTime=2017-04-28T11:14:23 SlurmdStartTime=2017-04-28T11:19:43
CapWatts=n/a
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
[[email protected] bacon]# munge -n | ssh compute-2-17 unmunge
STATUS: Success (0)
ENCODE_HOST: login.avi.hpc.uwm.edu <http://login.avi.hpc.uwm.edu>
(129.89.58.18)
ENCODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436)
DECODE_TIME: 2017-04-28 11:20:36 -0500 (1493396436)
TTL: 300
CIPHER: aes128 (4)
MAC: sha1 (3)
ZIP: none (0)
UID: root (0)
GID: root (0)
LENGTH: 0
[[email protected] bacon]# munge -n | ssh compute-2-18 unmunge
STATUS: Success (0)
ENCODE_HOST: login.avi.hpc.uwm.edu <http://login.avi.hpc.uwm.edu>
(129.89.58.18)
ENCODE_TIME: 2017-04-28 11:20:39 -0500 (1493396439)
DECODE_TIME: 2017-04-28 11:20:40 -0500 (1493396440)
TTL: 300
CIPHER: aes128 (4)
MAC: sha1 (3)
ZIP: none (0)
UID: root (0)
GID: root (0)
LENGTH: 0
[[email protected] bacon]# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up infinite 8 drng compute-5-[07-14]
batch* up infinite 29 mix
compute-2-[03-14],compute-5-[15-28,31,33-34]
batch* up infinite 66 alloc
compute-1-[01-05],compute-2-[19-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32]
batch* up infinite 37 idle
compute-1-[06-36],compute-2-[01-02,15-18]
batch-nice up infinite 8 drng compute-5-[07-14]
batch-nice up infinite 29 mix
compute-2-[03-14],compute-5-[15-28,31,33-34]
batch-nice up infinite 66 alloc
compute-1-[01-05],compute-2-[19-36],compute-3-[01-20],compute-4-[01-14],compute-5-[01-06,29-30,32]
batch-nice up infinite 37 idle
compute-1-[06-36],compute-2-[01-02,15-18]
highmem up infinite 2 idle compute-5-[35-36]
highmem-nice up infinite 2 idle compute-5-[35-36]
[[email protected] bacon]# srun --nodelist=compute-2-17 hostname
compute-2-17.avi
[[email protected] bacon]# srun --nodelist=compute-2-18 hostname
srun: error: Task launch for 360081.0 failed on node compute-2-18:
Communication connection failure
srun: error: Application launch failed: Communication connection
failure
srun: Job step aborted: Waiting up to 32 seconds for job step to
finish.
/var/log/slurm/slurmctld:
[2017-04-28T11:25:09.002] job_step_signal step 360081.0 not found
[2017-04-28T11:25:09.006] job_complete: JobID=360081 State=0x1
NodeCnt=1 WTERMSIG 105
[2017-04-28T11:25:09.054] job_complete: JobID=360081 State=0x8003
NodeCnt=1 done
[2017-04-28T11:25:09.054] agent/is_node_resp: node:compute-2-18
RPC:REQUEST_TERMINATE_JOB : Communication connection failure
[2017-04-28T11:25:42.153] error: Nodes compute-2-18 not responding
--
Earth is a beta site.
--
Earth is a beta site.
