[slurm-users] Job Invalid Account

We installed slurm 23.11.5 and we are receiving "JobId=n has invalid account" for every sbatch job. We are not using the slurm accounting/user database; we are using uniform UIDs and GIDs across the cluster. The jobs run and complete; can these invalid account errors be ignored or silenced? Job S

Re: [slurm-users] slurmctld removing offline nodes

rom:* slurm-users on behalf of > Joe Teumer > *Sent:* Tuesday, October 25, 2022 7:42:16 PM > *To:* slurm-us...@schedmd.com > *Subject:* [slurm-users] slurmctld removing offline nodes > > We noticed that the slurm controller will remove nodes that it cannot > reach. > How

[slurm-users] slurmctld removing offline nodes

We noticed that the slurm controller will remove nodes that it cannot reach. How can this be disabled? We would like to see the nodes marked down/drain instead of the controller removing the nodes from sinfo. /var/log/slurm/slurmctld.log [2022-10-25T13:10:01.500] debug: Log file re-opened [2022-1

Re: [slurm-users] slurm prolog script

You might be interested in using the PrologSlurmctld script instead of Prolog. Slurm Workload Manager - Prolog and Epilog Guide (schedmd.com) On Thu, Sep 15, 2022 at 2:08 PM Hoot Thompson wrote: > Can the prolog script be configured to only run on a

Re: [slurm-users] SUG22?

The site hasn't been updated. Slurm Workload Manager - Meetings (schedmd.com) >From last year: expect a few hours of video broadcast on their youtube channel; you can ask questions in chat 10 AM - 1 PM central Tuesday, September 20th? https://www.youtube.c

Re: [slurm-users] Intel MPI issue with slurm sbatch

Fixed with: Hydra Environment Variables (intel.com) <https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/hydra-environment-variables.html> I_MPI_HYDRA_BOOTSTRAP=ssh On Tue, Aug 16, 2022 at 11:09 AM Joe Teumer

[slurm-users] Intel MPI issue with slurm sbatch

Hello! Is there a way to turn off slurm MPI hooks? A job submitted via sbatch executes Intel MPI and the thread affinity settings are incorrect. However, running MPI manually over SSH works and all bindings are correct. We are looking to run our MPI jobs via slurm sbatch and have the same behavio

Re: [slurm-users] How to checkout a slurm node?

BIOS, running jobs via the KVM or SSH or SRUN What kind of BIOS settings would a user need to change? 1. F Clock, U Clock, Mem Clock, C states, Virtualization settings, and much more... On Fri, Nov 12, 2021 at 4:00 PM Joe Teumer wrote: > Hello! > > How best for a user to che

[slurm-users] How to checkout a slurm node?

Hello! How best for a user to check out a slurm node? Unfortunately, command 'salloc' doesn't appear to meet this need. Command `salloc --nodelist some_node --time 3:00:00` This gives the user a new shell and the user can use `srun` to start an interactive session. However, if the user needs to

[slurm-users] Possible bug with Prologslurmctld and Epilogslurmctld scripts?

Should the Prologslurmctld script only run after the Epilogslurmctld script finishes? Below you can see JobA runs and completes. While Epilogslurmctld (from JobA Node A) is executing on the Slurm controller the Prologslurmctld script for the next job (from Job B Node A) is also running on the Slur

[slurm-users] Prologslurmctld env variable issue

Apologies, I was not able to reply to the previous thread.[slurm-users] Prologslurmctld environment variables (google.com) I was not able to use "spank_job_control_setenv" successfully with sbatch. https://slurm.schedmd.com/spank.html I'm us

[slurm-users] Prologslurmctld environment variables

In the Prologslurmctld script environment there are only SLURM_* variables available. Is there any way to export additional variables to Prologslurmctld from an SBATCH job? I tried exporting variables in the SBATCH script, however, these variables do not get exported to Prologslurmctld, only to t