SLURM has worked this way as long as I can remember. If you don't use scontrol reboot_nodes, nodes are "down" when they come back because SLURM wasn't notified about the reboot. This is configurable in slurm.conf. ________________________________________ From: [email protected] <[email protected]> Sent: Tuesday, May 16, 2017 10:32:14 AM To: slurm-dev Subject: [slurm-dev] KNL node down after reboot
Hi, We want to introduce Intel Knights Landing (KNL) nodes into our cluster, and observed the following problem: Node reboots successfully with desired NUMA and MCDRAM modes, but remains in state down. From slurmctl.log: (…)[ 2017-05-16T15:16:10.437] _update_node_avail_features: nodes knlnode01 available features set to: a2a,hemi,quad,snc2,snc4,cache,flat,hybrid,auto,knights_landing [2017-05-16T15:16:10.437] _update_node_active_features: nodes knlnode01 active features set to: a2a,flat,knights_landing [2017-05-16T15:16:10.437] Node knlnode01 now responding [2017-05-16T15:16:10.437] validate_node_specs: Node knlnode01 unexpectedly rebooted boot_time=1494940521 last response=1494940189 [2017-05-16T15:16:10.437] requeue job 21433841 due to failure of node knlnode01 [2017-05-16T15:16:10.437] Requeuing JobID=21433841 State=0x0 NodeCnt=0 (…) From slurmd.log: (…) [2017-05-16T15:16:07.152] CPUs=256 Boards=1 Sockets=1 Cores=64 Threads=4 Memory=112514 TmpDisk=174986 Uptime=49 CPUSpecList=(null) FeaturesAvail=a2a,hemi,quad,snc2,snc4,cache,flat,hybrid,auto FeaturesActive=a2a,flat [2017-05-16T15:18:05.944] syscfg /d BIOSSETTINGS Cluster Mode [2017-05-16T15:18:05.944] Cluster Mode ============ Current Value : All2All ----------------------- Possible Values --------------- All2All : 00 SNC-2 : 01 SNC-4 : 02 Hemisphere : 03 Quadrant : 04 Auto : 05 [2017-05-16T15:18:06.158] syscfg /d BIOSSETTINGS Memory Mode [2017-05-16T15:18:06.158] Memory Mode =========== Current Value : Flat -------------------- Possible Values --------------- Cache : 00 Flat : 01 Hybrid : 02 Auto : 03 (…) Here some additional information: Slurm version: 17.02.2 (slurmctld and slurmd) slurm.conf: (…) # KNL NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures RebootProgram=/sbin/reboot (…) NodeName=DEFAULT CPUs=256 RealMemory=109568 Sockets=1 CoresPerSocket=64 ThreadsPerCore=4 TmpDisk=172032 NodeName=knlnode01 NodeAddr=10.1.12.1 Feature=knights_landing State=UNKNOWN (…) knl_generic.conf: SyscfgPath=/usr/bin/syscfg/syscfg DefaultNUMA=a2a AllowNUMA=a2a,snc2 DefaultMCDRAM=cache sbatch Command: sbatch -p phi -C flat,a2a -N1 --exclusive job.slurm Any suggestion is welcome. Cheers, Nico
