Dear list
We have a relatively new installation of SLURM. We have started to have a problem with some of the nodes when using srun [scott@cream-ce ~]$ srun --pty -w n38 hostname srun: error: fwd_tree_thread: can't find address for host n38, check slurm.conf srun: error: Task launch for 18710.0 failed on node n38: Can't find an address, check slurm.conf srun: error: Application launch failed: Can't find an address, check slurm.conf srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: error: Timed out waiting for job step to complete I’ve spent most of the day trying to follow others who’ve had similar problems and check everything but I haven’t made any progress — Using sbatch there is no problem: jobs launch on the given node and finish normally and reliably — srun works fine for most of the nodes — the slum.conf file is identical on all nodes (checked by diffing and no complaints in the logs) — both the slurmctld and the slurmd start cleanly with no obvious errors or warnings (e.g. about slurm.conf) — sinfo reports that all our nodes are up, some busy some not. The problem is independent of load on the nodes — I’ve increased the log level on the control daemon and there’s no obvious additional information when the srun happens — we use puppet to maintain our infrastructure so while there must be a difference between the machines that work and those that don’t I can see it. — all nodes run ntpd and the times appear the same when checked manually — all nodes have plenty of disk space — I’ve tried restarting both the slurm and control daemons and this has no effect, even for a short time — hostname on working and problematic nodes give the expected results in the same format as other — all hostnames are in /etc/hosts on all machines — we currently have just less than 40 worker nodes, treewidth=50 We’re running SLURM 17.11.10 under CentOS 7.5 This is the final part of the slurm.conf file NodeName=n[02,08,10,29-40,42-45] RealMemory=131072 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN NodeName=n[05] RealMemory=256000 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN NodeName=n[07] RealMemory=45000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN NodeName=n[15] RealMemory=48000 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN NodeName=n[16] RealMemory=31000 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN NodeName=n[17] RealMemory=215000 Sockets=16 CoresPerSocket=8 ThreadsPerCore=1 State=UNKNOWN NodeName=n[18] RealMemory=90000 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN NodeName=n[19] RealMemory=515000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN NodeName=n[20] RealMemory=64000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN NodeName=n[21] RealMemory=64000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN NodeName=n[22] RealMemory=56000 Sockets=16 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN NodeName=n[23] RealMemory=225500 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN NodeName=n[27] RealMemory=65536 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN NodeName=n[28] RealMemory=65536 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN PartitionName=batch Nodes=n[02,05,07-08,10,15-23,27-40,42-45] Default=YES MaxTime=40320 State=UP As examples — fails: n15, n17, n27, n28, n38, n45 — success: n02, n10, n16, n18, n29 onwards except for n38, n45 Many thanks for any help Scott This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary.
