Thank you for the input! @Ryan I didn’t’ check the code of the knl_generic plugin, but I would expect that in this case it should not represent an unexpected reboot from Slurm’s point of view.
@Costin Increasing ResumeTimeout in slurm.conf worked. Although slurmd is still down after reboot, now with reason “low memory” instead of “unexpected reboot”. But that’s a different problem. Cheers, Nico Von: Costin Caramarcu <[email protected]> Antworten an: slurm-dev <[email protected]> Datum: Dienstag, 16. Mai 2017 um 16:59 An: slurm-dev <[email protected]> Betreff: [slurm-dev] Re: KNL node down after reboot Hi, A few suggestions: 1) Try increasing the timeouts: SlurmctldTimeout=600 SlurmdTimeout=600 ResumeTimeout=600 2) Make sure that when slurm starts the node finished mounting file systems and the whole boot procedure is done, Regards, Costin On Tue, May 16, 2017 at 10:40 AM, Ryan Novosielski <[email protected]<mailto:[email protected]>> wrote: SLURM has worked this way as long as I can remember. If you don't use scontrol reboot_nodes, nodes are "down" when they come back because SLURM wasn't notified about the reboot. This is configurable in slurm.conf. ________________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Sent: Tuesday, May 16, 2017 10:32:14 AM To: slurm-dev Subject: [slurm-dev] KNL node down after reboot Hi, We want to introduce Intel Knights Landing (KNL) nodes into our cluster, and observed the following problem: Node reboots successfully with desired NUMA and MCDRAM modes, but remains in state down. From slurmctl.log: (…)[ 2017-05-16T15:16:10.437] _update_node_avail_features: nodes knlnode01 available features set to: a2a,hemi,quad,snc2,snc4,cache,flat,hybrid,auto,knights_landing [2017-05-16T15:16:10.437] _update_node_active_features: nodes knlnode01 active features set to: a2a,flat,knights_landing [2017-05-16T15:16:10.437] Node knlnode01 now responding [2017-05-16T15:16:10.437] validate_node_specs: Node knlnode01 unexpectedly rebooted boot_time=1494940521 last response=1494940189 [2017-05-16T15:16:10.437] requeue job 21433841 due to failure of node knlnode01 [2017-05-16T15:16:10.437] Requeuing JobID=21433841 State=0x0 NodeCnt=0 (…) From slurmd.log: (…) [2017-05-16T15:16:07.152] CPUs=256 Boards=1 Sockets=1 Cores=64 Threads=4 Memory=112514 TmpDisk=174986 Uptime=49 CPUSpecList=(null) FeaturesAvail=a2a,hemi,quad,snc2,snc4,cache,flat,hybrid,auto FeaturesActive=a2a,flat [2017-05-16T15:18:05.944] syscfg /d BIOSSETTINGS Cluster Mode [2017-05-16T15:18:05.944] Cluster Mode ============ Current Value : All2All ----------------------- Possible Values --------------- All2All : 00 SNC-2 : 01 SNC-4 : 02 Hemisphere : 03 Quadrant : 04 Auto : 05 [2017-05-16T15:18:06.158] syscfg /d BIOSSETTINGS Memory Mode [2017-05-16T15:18:06.158] Memory Mode =========== Current Value : Flat -------------------- Possible Values --------------- Cache : 00 Flat : 01 Hybrid : 02 Auto : 03 (…) Here some additional information: Slurm version: 17.02.2 (slurmctld and slurmd) slurm.conf: (…) # KNL NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures RebootProgram=/sbin/reboot (…) NodeName=DEFAULT CPUs=256 RealMemory=109568 Sockets=1 CoresPerSocket=64 ThreadsPerCore=4 TmpDisk=172032 NodeName=knlnode01 NodeAddr=10.1.12.1 Feature=knights_landing State=UNKNOWN (…) knl_generic.conf: SyscfgPath=/usr/bin/syscfg/syscfg DefaultNUMA=a2a AllowNUMA=a2a,snc2 DefaultMCDRAM=cache sbatch Command: sbatch -p phi -C flat,a2a -N1 --exclusive job.slurm Any suggestion is welcome. Cheers, Nico
