Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-05 Thread Marcus Wagner
-users On Behalf Of Brian W. Johanson Sent: Tuesday, February 4, 2020 1:35 PM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu Please include the output for: scontrol show node=liqidos-dean-node1 scontrol show part

Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-04 Thread dean.w.schulze
@lists.schedmd.com Subject: Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu Please include the output for: scontrol show node=liqidos-dean-node1 scontrol show partition=Partition_you_are_attempting_to_submit_to and any other #SBATCH lines submitted with the failin

Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-04 Thread Brian W. Johanson
md on all nodes. Still get the same problem. -Original Message- From: slurm-users On Behalf Of Marcus Wagner Sent: Tuesday, February 4, 2020 2:31 AM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu Hi De

Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-04 Thread dean.w.schulze
I've already restarted slurmctld and slurmd on all nodes. Still get the same problem. -Original Message- From: slurm-users On Behalf Of Marcus Wagner Sent: Tuesday, February 4, 2020 2:31 AM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] sbatch script won't acc

Re: [slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-04 Thread Marcus Wagner
Hi Dean, could you please try to restart the slurmctld? This usually helps on our site. Never saw this with gres happening, but many other times. This is, why we restart slurmctld once a day by a cron job. Best Marcus On 2/4/20 12:59 AM, Dean Schulze wrote: When I run an sbatch script with t

[slurm-users] sbatch script won't accept --gres that requires more than 1 gpu

2020-02-03 Thread Dean Schulze
When I run an sbatch script with the line #SBATCH --gres=gpu:gp100:1 it runs. When I change it to #SBATCH --gres=gpu:gp100:3 it fails with "Requested node configuration is not available". But I have a node with 4 gp100s available. Here's my slurm.conf: NodeName=liqidos-dean-node1 CPUs=2 Boa