Hi,
A few suggestions:
1) Try increasing the timeouts:
SlurmctldTimeout=600
SlurmdTimeout=600
ResumeTimeout=600
2) Make sure that when slurm starts the node finished mounting file
systems and the whole boot procedure is done,
Regards,
Costin
On Tue, May 16, 2017 at 10:40 AM, Ryan Novosielski <[email protected]>
wrote:
>
> SLURM has worked this way as long as I can remember. If you don't use
> scontrol reboot_nodes, nodes are "down" when they come back because SLURM
> wasn't notified about the reboot. This is configurable in slurm.conf.
> ________________________________________
> From: [email protected] <[email protected]>
> Sent: Tuesday, May 16, 2017 10:32:14 AM
> To: slurm-dev
> Subject: [slurm-dev] KNL node down after reboot
>
> Hi,
>
> We want to introduce Intel Knights Landing (KNL) nodes into our cluster,
> and observed the following problem: Node reboots successfully with desired
> NUMA and MCDRAM modes, but remains in state down.
>
> From slurmctl.log:
>
> (…)[
> 2017-05-16T15:16:10.437] _update_node_avail_features: nodes knlnode01
> available features set to: a2a,hemi,quad,snc2,snc4,cache,
> flat,hybrid,auto,knights_landing
> [2017-05-16T15:16:10.437] _update_node_active_features: nodes knlnode01
> active features set to: a2a,flat,knights_landing
> [2017-05-16T15:16:10.437] Node knlnode01 now responding
> [2017-05-16T15:16:10.437] validate_node_specs: Node knlnode01 unexpectedly
> rebooted boot_time=1494940521 last response=1494940189
> [2017-05-16T15:16:10.437] requeue job 21433841 due to failure of node
> knlnode01
> [2017-05-16T15:16:10.437] Requeuing JobID=21433841 State=0x0 NodeCnt=0
> (…)
>
> From slurmd.log:
>
> (…)
> [2017-05-16T15:16:07.152] CPUs=256 Boards=1 Sockets=1 Cores=64 Threads=4
> Memory=112514 TmpDisk=174986 Uptime=49 CPUSpecList=(null)
> FeaturesAvail=a2a,hemi,quad,snc2,snc4,cache,flat,hybrid,auto
> FeaturesActive=a2a,flat
> [2017-05-16T15:18:05.944] syscfg /d BIOSSETTINGS Cluster Mode
> [2017-05-16T15:18:05.944]
> Cluster Mode
> ============
> Current Value : All2All
> -----------------------
> Possible Values
> ---------------
> All2All : 00
> SNC-2 : 01
> SNC-4 : 02
> Hemisphere : 03
> Quadrant : 04
> Auto : 05
>
> [2017-05-16T15:18:06.158] syscfg /d BIOSSETTINGS Memory Mode
> [2017-05-16T15:18:06.158]
> Memory Mode
> ===========
> Current Value : Flat
> --------------------
> Possible Values
> ---------------
> Cache : 00
> Flat : 01
> Hybrid : 02
> Auto : 03
> (…)
>
>
> Here some additional information:
>
> Slurm version: 17.02.2 (slurmctld and slurmd)
>
> slurm.conf:
> (…)
> # KNL
> NodeFeaturesPlugins=knl_generic
> DebugFlags=NodeFeatures
> RebootProgram=/sbin/reboot
> (…)
> NodeName=DEFAULT CPUs=256 RealMemory=109568 Sockets=1 CoresPerSocket=64
> ThreadsPerCore=4 TmpDisk=172032
> NodeName=knlnode01 NodeAddr=10.1.12.1 Feature=knights_landing State=UNKNOWN
> (…)
>
> knl_generic.conf:
> SyscfgPath=/usr/bin/syscfg/syscfg
> DefaultNUMA=a2a
> AllowNUMA=a2a,snc2
> DefaultMCDRAM=cache
>
>
> sbatch Command:
> sbatch -p phi -C flat,a2a -N1 --exclusive job.slurm
>
>
>
> Any suggestion is welcome.
>
> Cheers,
> Nico
>
