[slurm-users] Resource assignment problems.

Hello, I'm running Slurm 15.08 and I'm having a problem I'm a bit confused about why it is happening. I have a user that is submitting asking for 64 tasks, the default system wide is 1 cpu per task. But the user is getting varied numbers of cpus for each task, sometimes as few as 2 sometimes al

[slurm-users] Disable Account Limits Per Partition?

Hi, I'm not sure of the best way to solve this and I don't see any obvious things I can set in the configuration. Please let me know if I'm missing something. I have several partitions in Slurm (16.05). I also have many accounts with users tied to them and all of the accounts have a CPU hour li

[slurm-users] Requesting total GPUs or memory, not per node.

Hello, I'm relatively new to administering slurm, so my apologies if I've missed something obvious. We have nodes of 4 GPU and nodes of 8 GPU. I would like users to be able to request a total number of GPUs they require. The MPI software is not fussed how many nodes it spans. I had hoped request

[slurm-users] Extreme long db upgrade 16.05.6 -> 17.11.3

Hello, We have been trying to upgrade slurm on our cluster from 16.05.6 to 17.11.3. I'm thinking this should be doable? Past upgrades have been a breeze, and I believe during the last one, the db upgrade took like 25 minutes. Well now, the db upgrade process is taking far too long. We previousl

Re: [slurm-users] Extreme long db upgrade 16.05.6 -> 17.11.3

On Wed, Feb 21, 2018 at 11:56:38PM +, Christopher Benjamin Coffey wrote: > Hello, > > We have been trying to upgrade slurm on our cluster from 16.05.6 to 17.11.3. > I'm thinking this should be doable? Past upgrades have been a breeze, and I > believe during the last one, the db upgrade took

Re: [slurm-users] Extreme long db upgrade 16.05.6 -> 17.11.3

This is great to know Kurt. We can't be the only folks running into this.. I wonder if the mysql update code gets into a deadlock or something. I'm hoping a slurm dev will chime in ... Kurt, out of band if need be, I'd be interested in the details of what you ended up doing. Best, Chris — Chr

Re: [slurm-users] Requesting total GPUs or memory, not per node.

Hey Rob, Perhaps something in the direction of srun --ntasks=2 --gres=gpu:4 nvidia-smi , help you? this will run two tasks each with 4 gpu and execute nvidia-smi, the output should be similar of doing nvidia-smi on one 8 gpu server On 22/02/2018 01:26, Rob

[slurm-users] ntasks and cpus-per-task

Hi all, I'm quite noob in slurm and I'm still learning how slurm works. There are some concepts that get me confused. For example, the difference between ntasks and cpus-per-task in sbatch and/or srun. I've noticed that cpus-per-task (and ntasks=1) allocates cpus (cores) within the same compute no