Thanks, nice workaround. It will be great if there was a way to actually set it so that one can use only one node per job, a bit like ---exclusive. Thanks
On Tue, 3 Dec 2024 at 16:24, Renfro, Michael <ren...@tntech.edu> wrote: > I’ve never done this myself, but others probably have. At the end of [1], > there’s an example of making a generic resource for bandwidth. You could > set that to any convenient units (bytes/second or bits/second, most > likely), and assign your nodes a certain amount. Then any network-intensive > job could reserve all the node’s bandwidth, without locking other > less-intensive jobs off the node. It’s identical to reserving 1 or more > GPUs per node, just without any hardware permissions. > > > > [1] https://slurm.schedmd.com/gres.conf.html#SECTION_EXAMPLES > > > > *From: *Oren <oren...@gmail.com> > *Date: *Tuesday, December 3, 2024 at 3:15 PM > *To: *Renfro, Michael <ren...@tntech.edu> > *Cc: *slurm-us...@schedmd.com <slurm-us...@schedmd.com> > *Subject: *Re: [slurm-users] How can I make sure my user have only one > job per node (Job array --exclusive=user,) > > *External Email Warning* > > *This email originated from outside the university. Please use caution > when opening attachments, clicking links, or responding to requests.* > ------------------------------ > > Thank you Michael, > > yeah, you guessed right, Networking. > My job is mostly IO (Networking) intensive, my nodes connect to the > network via a non blocking switch, but the ethernet cards are not the best, > > So I don't need many CPUs per node, but I do want to run on all nodes to > fully utilize the network connection that each node has. > > > > Assuming I don't want to change the scheduler, is there anything else I > can do? > > Thanks, > > Oren > > > > On Tue, 3 Dec 2024 at 15:10, Renfro, Michael <ren...@tntech.edu> wrote: > > I’ll start with the question of “why spread the jobs out more than > required?” and move on to why the other items didn’t work: > > > > 1. exclusive only ensures that others’ jobs don’t run on a node with > your jobs, and does nothing about other jobs you own. > 2. spread-job distributes the work of one job across multiple nodes, > but does nothing about multiple jobs > 3. distribution also distributes the work of one job > > > > You might get something similar to what you want by changing the scheduler > to use CR_LLN instead of CR_Core_Memory (or whatever you’re using), but > that’ll potentially have serious side effects for others’ jobs. > > > > So back to the original question: why **not** pack 20 jobs onto fewer > nodes if those nodes have the capacity to run the full set of jobs? You > shouldn’t have a constraint with memory or CPUs. Are you trying to spread > out an I/O load somehow? Networking? > > > > *From: *Oren via slurm-users <slurm-users@lists.schedmd.com> > *Date: *Tuesday, December 3, 2024 at 1:35 PM > *To: *slurm-us...@schedmd.com <slurm-us...@schedmd.com> > *Subject: *[slurm-users] How can I make sure my user have only one job > per node (Job array --exclusive=user,) > > *External Email Warning* > > *This email originated from outside the university. Please use caution > when opening attachments, clicking links, or responding to requests.* > ------------------------------ > > Hi, > I have a cluster of 20-nodes, and I want to run a jobarray on that > cluster, but I want each node to get one job per node. > > > > When I do the following: > > #!/bin/bash > > #SBATCH --job-name=process_images_train # Job name > > #SBATCH --time=50:00:00 # Time limit hrs:min:sec > > #SBATCH --tasks=1 > > #SBATCH --cpus-per-task=4 > > #SBATCH --mem=50000 > > #SBATCH --array=0-19# 19 # Job array with 20 jobs (0 to 19) > > > > I get 10 jobs in node #1 and 10 jobs in node #20, I want a job in each > node. > > > > I've tried: > #SBATCH --exclusive=user > > Also > > #SBATCH --spread-job > > #SBATCH --distribution=cyclic > > > > > > Nothing changes, node#1 got 10 jobs and node#2 got 10 jobs. > > Thanks > >
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
