Re: [slurm-users] Incorrect Number of GPUs?

2021-07-26 Thread Fulcomer, Samuel
Yeah, you'd think after all this time it would, bu it remains a bit of arcane knowledge that's mostly passed on in oral history There are some things that the slurmd processes need to be restarted for, as well. I have a vague memory that changing the debug level is one... On Mon, Jul 26, 2021

Re: [slurm-users] Incorrect Number of GPUs?

2021-07-26 Thread Jason Simms
Dear Samuel, Restarting slurmctld did the trick. Thanks! I should have thought to do that, but typically sconrtrol reconfigure picks up most changes. Warmest regards, Jason On Mon, Jul 26, 2021 at 12:55 PM Fulcomer, Samuel wrote: > ...and... you need to restart slurmctld when you change a Node

Re: [slurm-users] Incorrect Number of GPUs?

2021-07-26 Thread Fulcomer, Samuel
...and... you need to restart slurmctld when you change a NodeName line. "scontrol reconfigure" doesn't do the truck. On Mon, Jul 26, 2021 at 12:49 PM Fulcomer, Samuel wrote: > If you have a dual-root PCIe system you may need to specify the CPU/core > affinity in gres.conf. > > On Mon, Jul 26, 2

Re: [slurm-users] Incorrect Number of GPUs?

2021-07-26 Thread Fulcomer, Samuel
If you have a dual-root PCIe system you may need to specify the CPU/core affinity in gres.conf. On Mon, Jul 26, 2021 at 12:07 PM Jason Simms wrote: > Hello all, > > I have a GPU node with 3 identical GPUs (we started with two and recently > added the third). Running nvidia-smi correctly shows th

[slurm-users] Incorrect Number of GPUs?

2021-07-26 Thread Jason Simms
Hello all, I have a GPU node with 3 identical GPUs (we started with two and recently added the third). Running nvidia-smi correctly shows that all three are recognized. My gres.conf file has only this line: NodeName=gpu01 File=/dev/nvidia[0-2] Type=quadro_8000 Name=gpu Count=3 And the relevant l

Re: [slurm-users] sreport getting last 30 reports

2021-07-26 Thread Killian Murphy
sreport user topusage start=$(date +"%Y-%m-%d" --date="-30 days") On Mon, 26 Jul 2021 at 16:48, Amjad Syed wrote: > Hello > I am trying to find what is the best way of using Start=now - 30 days in > sreport > > I have used sreport User TopUsage Start=now-30Days > But it does not seem to take it.

[slurm-users] sreport getting last 30 reports

2021-07-26 Thread Amjad Syed
Hello I am trying to find what is the best way of using Start=now - 30 days in sreport I have used sreport User TopUsage Start=now-30Days But it does not seem to take it. Any thing i am missing here? Majid