Dear all,
I stumbled upon an issue that might be a bug, config error or a bad submit
script. Running on 16.05.9 (but occured also on
16.05.7) I have two partitions configured, one for exclusive usage, the other
for shared but user exclusive usage:
PartitionName=TS \
Nodes=ALL \
Default=YES \
MaxTime=70-00:00:00 \
State=UP \
OverSubscribe=EXCLUSIVE
PartitionName=TS-shared \
Nodes=ALL \
MaxTime=70-00:00:00 \
State=UP \
OverSubscribe=FORCE \
ExclusiveUser=YES
All nodes have 20 cores (40 threads) and 256GB memory.
To limit user groups I have configured GrpTRES on account level. No QoS is
configured for the user that experiences the mentioned
issue.
The issue is as follows. The user submits array jobs using the following script:
-------------------------------
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH -A simprod
#SBATCH -t 21-00:00:00
#SBATCH --partition=TS-shared
#SBATCH --oversubscribe
#SBATCH --mem-per-cpu=1000
#SBATCH --array=1-13
# run x copies of the same sequential program
# each program puts its outputfiles in a separate directory
##############################################################
cd $SLURM_ARRAY_TASK_ID
./run_cmd
-------------------------------
As you can see, she's asking for 13 subjobs, that should be able to run on one
node in parallel. But it happens that only one
subnode is started and all others are PENDING with reason AssocGrpNodeLimit.
There also has been the situation where she was running an array job where
about 6 subjobs were currently running on 2 nodes but
Slurm was incorrectly counting the usage as 6 nodes and other jobs were
therefore stuck in PENDING with reason AssocGrpNodeLimit.
In both situations there were jobs of other users of the same group running in
the exclusive partition.
Questions:
* Does the submit script look correct or should she use "srun ./run_cmd"?
* Is the cluster configuration correct in regard to shared usage? If not, what
changes are recommended?
* Is this a bug in Slurm where it incorrectly counts shared nodes multiple
times in regard to configured group node limits?
* Any other explanation for this behavior?
Regards,
Uwe
