David, I'd be curious to know if the device files exist (`ls /dev/nvidia*`). On our systems with GPUs, we've seen this error when the /dev/nvidia? files are missing because either a GPU has failed or the CUDA drivers not being installed appropriately.
- Jared -----Original Message----- From: David van Leeuwen [mailto:[email protected]] Sent: Tuesday, December 6, 2016 7:22 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] Re: Reason=gres/gpu count too low Hi, I have restarted the slurmctld and slurmd several times---in fact, I don't know what the preferred order of restart is (I suppose first slurmd and then slurmctld). But all to no avail. Is there a way I can somehow completely re-init the status of slurm? I've tried "service slurmd startclean" (after fixing a debian typo)---but I keep getting this status Reason=gres/gpu count too low (0 < 2) [root@2016-12-02T15:16:44] with an old date. Thanks, ---david On Tue, Dec 6, 2016 at 2:07 PM, Robbert Eggermont <[email protected]> wrote: > > On 06-12-16 10:49, David van Leeuwen wrote: >> >> "gres/gpu count too low (0 < 1)" > > > Last time I saw this I had to restart the slurmd on that node (a > simple scontrol reconfigure was not enough). > > I guess this message indicates a discrepancy between the number of GPU > resources detected by slurmd at startup, and the number specified in > the slurm.conf (and used by slurmctld). > > In the end, I even had to restart both slurmd and slurmctld to get the > GPUs registered properly (including the CPU specification in the > gres.conf). (And I then repeated this just in case the order was > important.;-)) > > Best, > > Robbert -- David van Leeuwen
