Looks like it is not doing what you think it should. It does state:
If the number of tasks is given and a number of requested nodes is also
given, the number of nodes used from that request will be reduced to
match that of the number of tasks if the number of nodes in the request
is greater than the number of tasks.
Your number of nodes was reduced because you only requested 3 tasks.
You are correct that it does not specify in detail what more than one
value would do beyond making it a range of min/max. The way I read it,
you can put multiple values and the other value(s) would be a max nodes.
The "size_string" looks like parsing code similar to that used for
arrays. Nowhere does it say what the effect of that is for --nodes.
You can either figure out what you want to actually do and do that, or
open a bug with schedmd to add some clarity to the documentation. They
are more than happy to do that.
Brian Andrus
On 8/29/2024 11:48 PM, Matteo Guglielmi via slurm-users wrote:
I'm sorry, but I still don't get it.
Isn't --nodes=2,4 telling slurm to allocate 2 OR 4 nodes and nothing else?
So, if:
--nodes=2 allocates only two nodes
--nodes=4 allocates only four nodes
--nodes=1-2 allocates min one and max two nodes
--nodes=1-4 allocates min one and max four nodes
what is the allocation rule for --nodes=2,4 which is the so-called size_string
allocation?
man sbatch says:
Node count can also be specified as size_string. The size_string specification
identifies what nodes
values should be used. Multiple values may be specified using a comma separated
list or with a step
function by suffix containing a colon and number values with a "-" separator.
For example, "--nodes=1-15:4" is equivalent to "--nodes=1,5,9,13".
...
The job will be allocated as many nodes as possible within the range specified
and without delaying the
initiation of the job.
________________________________
From: Brian Andrus via slurm-users<slurm-users@lists.schedmd.com>
Sent: Thursday, August 29, 2024 7:27:44 PM
To:slurm-users@lists.schedmd.com
Subject: [slurm-users] Re: playing with --nodes=<size_string>
It looks to me that you requested 3 tasks spread across 2 to 4 nodes. Realize
--nodes is not targeting your nodes named 2 and 4, it is a count of how many
nodes to use. You only needed 3 tasks/cpus, so that is what you were allocated
and you have 1 cpu per node, so you get 3 (of up to 4) nodes. Slurm does not
give you 4 nodes because you only want 3 tasks.
You see the result in your variables:
SLURM_NNODES=3
SLURM_JOB_CPUS_PER_NODE=1(x3)
If you only want 2 nodes, make --nodes=2.
Brian Andrus
On 8/29/24 08:00, Matteo Guglielmi via slurm-users wrote:
Hi,
On sbatch's manpage there is this example for <size_string>:
--nodes=1,5,9,13
so either one specifies <minnodes>[-maxnodes] OR <size_string>.
I checked the logs, and there are no reported errors about wrong or ignored
options.
MG
________________________________
From: Brian Andrus via
slurm-users<slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com>
Sent: Thursday, August 29, 2024 4:11:25 PM
To:slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com>
Subject: [slurm-users] Re: playing with --nodes=<size_string>
Your --nodes line is incorrect:
-N, --nodes=<minnodes>[-maxnodes]|<size_string>
Request that a minimum of minnodes nodes be allocated to this job. A maximum
node count may also be specified with maxnodes.
Looks like it ignored that and used ntasks with ntasks-per-node as 1, giving
you 3 nodes. Check your logs and check your conf see what your defaults are.
Brian Andrus
On 8/29/2024 5:04 AM, Matteo Guglielmi via slurm-users wrote:
Hello,
I have a cluster with four Intel nodes (node[01-04], Feature=intel) and four
Amd nodes (node[05-08], Feature=amd).
# job file
#SBATCH --ntasks=3
#SBATCH --nodes=2,4
#SBATCH --constraint="[intel|amd]"
env | grep SLURM
# slurm.conf
PartitionName=DEFAULT MinNodes=1 MaxNodes=UNLIMITED
# log
SLURM_JOB_USER=software
SLURM_TASKS_PER_NODE=1(x3)
SLURM_JOB_UID=1002
SLURM_TASK_PID=49987
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/home/software
SLURMD_NODENAME=node01
SLURM_JOB_START_TIME=1724932865
SLURM_CLUSTER_NAME=cluster
SLURM_JOB_END_TIME=1724933465
SLURM_CPUS_ON_NODE=1
SLURM_JOB_CPUS_PER_NODE=1(x3)
SLURM_GTIDS=0
SLURM_JOB_PARTITION=nodes
SLURM_JOB_NUM_NODES=3
SLURM_JOBID=26
SLURM_JOB_QOS=lprio
SLURM_PROCID=0
SLURM_NTASKS=3
SLURM_TOPOLOGY_ADDR=node01
SLURM_TOPOLOGY_ADDR_PATTERN=node
SLURM_MEM_PER_CPU=0
SLURM_NODELIST=node[01-03]
SLURM_JOB_ACCOUNT=dalco
SLURM_PRIO_PROCESS=0
SLURM_NPROCS=3
SLURM_NNODES=3
SLURM_SUBMIT_HOST=master
SLURM_JOB_ID=26
SLURM_NODEID=0
SLURM_CONF=/etc/slurm/slurm.conf
SLURM_JOB_NAME=mpijob
SLURM_JOB_GID=1002
SLURM_JOB_NODELIST=node[01-03] <<<=== why three nodes? Shouldn't this still be
two nodes?
Thank you.
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
