One thing, I forgot.
On 8/20/19 4:58 PM, Christopher Benjamin Coffey wrote:
Hi Marcus,
What is the reason to add "--mem-per-cpu" when the job already has exclusive
access to the node?
The user (normally) does not set --exclusive directly. We have several
accounts, whose jobs by default should run exclusively, so we set that
in the job_submit plugin.
Your job has access to all of the memory, and all of the cores on the system
already. Also note, for non-mpi code like single core job, or shared memory
threaded job, you want to ask for number of cpus with --cpus-per-task, or -c.
Unless you are running mpi code, where you will want to use -n, and --ntasks
instead to launch n copies of the code on n cores. In this case, because you
asked for -n2, and also specified a mem-per-cpu request, the scheduler is
doling out the memory as requested (2 x tasks), likely due to having
SelectTypeParameters=CR_Core_Memory in slurm.conf.
I must say, we would be much happier with a --mem-per-task option
instead. I still do not know, why one should ask for mem-per-cpu
(logically), since in a shared memory job, you start one process, the
threads share the memory.
With an hybrid MPI-code (mpi code with openmp parallelization on the
tasks), it makes even less sense. If I know, how much memory my tasks
needs, e.g. 10 GB, I still have to divide that through the number of
threads (-c) to get the right memory request. For me as an
administrator, an openmp job is a special hybrid job with only one
requested task. So it is the same for a shared memory job. I always have
to divide the really needed memory through the number of threads (or
cpus-per-task).
Is there anyone, who can enlighten me?
Why does one have to ask for memory per smallest scheduleable (is that
word right?) unit? Isn't it better to ask for memory per task/process?
Best
Marcus
Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
On 8/20/19, 1:37 AM, "slurm-users on behalf of Marcus Wagner"
<slurm-users-boun...@lists.schedmd.com on behalf of wag...@itc.rwth-aachen.de> wrote:
Just made another test.
Thanks god, the exclusivity is not "destroyed" completely, only on job
can run on the node, when the job is exclusive. Nonetheless, this is
somewhat unintuitive.
I wonder, if that also has an influence on the cgroups and the process
affinity/binding.
I will do some more tests.
Best
Marcus
On 8/20/19 9:47 AM, Marcus Wagner wrote:
> Hi Folks,
>
>
> I think, I've stumbled over a BUG in Slurm regarding the
> exclusiveness. Might also, I've misinterpreted something. I would be
> happy, if someone could explain that to me in the latter case.
>
> To the background. I have set PriorityFlags=MAX_TRES
> The TRESBillingWeights are "CPU=1.0,Mem=0.1875G" for a partition with
> 48 core nodes and RealMemory 187200.
>
> ---
>
> I have two jobs:
>
> job 1:
> #SBATCH --exclusive
> #SBATCH --ntasks=2
> #SBATCH --nodes=1
>
> scontrol show <jobid> =>
> NumNodes=1 NumCPUs=48 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> TRES=cpu=48,mem=187200M,node=1,billing=48
>
> exactly, what I expected, I got 48 CPUs and therefore the billing is 48.
>
> ---
>
> job 2 (just added mem-per-cpu):
> #SBATCH --exclusive
> #SBATCH --ntasks=2
> #SBATCH --nodes=1
> #SBATCH --mem-per-cpu=5000
>
> scontrol show <jobid> =>
> NumNodes=1-1 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> TRES=cpu=2,mem=10000M,node=1,billing=2
>
> Why "destroys" '--mem-per-cpu' exclusivity?
>
>
>
> Best
> Marcus
>
--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wag...@itc.rwth-aachen.de
https://nam05.safelinks.protection.outlook.com/?url=www.itc.rwth-aachen.de&data=02%7C01%7Cchris.coffey%40nau.edu%7C4a5803448abd497d7cde08d7254995f2%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C637018870287848104&sdata=HNuqCBYwrJjBcLGFGYuVKxWe9pqCxt028rrRrJ%2FTYp0%3D&reserved=0
--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wag...@itc.rwth-aachen.de
www.itc.rwth-aachen.de
