Hi Shunran,
we do something very similar. I have nodes with 2 SSDs in a Raid1
mounted on /local. We defined a gres ressource just like you and called
it local. We define the ressource in the gres.conf like this:
# LOCAL
NodeName=hpc-node[01-10] Name=local
and add the ressource in counts of GB to the slurm.nodes.conf:
NodeName=hpc-node01 CPUs=256 RealMemory=... Gres=local:3370
So in this case the node01 has 3370 counts or GB of the gres "local"
available for reservation. Now slurm tracks that resource for you and
users can reserve counts of /local space. But there is still one big
problem, SLURM hast no idea what local is and as u correctly noted,
others can just use it. I solved this the following way:
- /local ist owned by root, so no user can just write to it
- the node prolog creates a folder in /local in this form:
/local/job_<SLURM_JOB_ID> and makes the job owner of it
- the node epilog deletes that folder
This way you have already solved the problem of people/jobs not having
reserved any local using it. But there ist still no enforcement of
limits. For that I use quotas.
My /local is XFS formatted and XFS has a nifty feature called project
quotas, where you can set the quota for a folder.
This is my node prolog script for this purpose:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
local_dir="/local"
local_job=0
## DETERMINE GRES:LOCAL
# get job gres
JOB_TRES=$(scontrol show JobID=${SLURM_JOBID} | grep "TresPerNode=" |
cut -d '=' -f 2 | tr ',' ' ')
# parse for local
for gres in ${JOB_TRES}; do
key=$(echo ${gres} | cut -d ':' -f 2 | tr '[:upper:]' '[:lower:]')
if [[ ${key} == "local" ]]; then
local_job=$(echo ${gres} | cut -d ':' -f 3)
break
fi
done
# make job local-dir if requested
if [[ ${local_job} -ne 0 ]]; then
# make local-dir for job
SLURM_TMPDIR="${local_dir}/job_${SLURM_JOBID}"
mkdir ${SLURM_TMPDIR}
# conversion
local_job=$((local_job * 1024 * 1024))
# set hard limit to requested size + 5%
hard_limit=$((local_job * 105 / 100))
# create project quota and set limits
xfs_quota -x -c "project -s -p ${SLURM_TMPDIR} ${SLURM_JOBID}"
${local_dir}
xfs_quota -x -c "limit -p bsoft=${local_job}k bhard=${hard_limit}k
${SLURM_JOBID}" ${local_dir}
chown ${SLURM_JOB_USER}:0 ${SLURM_TMPDIR}
chmod 750 ${SLURM_TMPDIR}
fi
exit 0
This is my epilog:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
local_dir="/local"
SLURM_TMPDIR="${local_dir}/job_${SLURM_JOBID}"
# remove the quota
xfs_quota -x -c "limit -p bsoft=0m bhard=0m ${SLURM_JOBID}"
${local_dir}
# remove the folder
if [[ -d ${SLURM_TMPDIR} ]]; then
rm -rf --one-file-system ${SLURM_TMPDIR}
fi
exit 0
In order to use project quota you would need to activate it by using
this mount flag: pquota in the fstab.
I give the user 5% more than he requested. You just have to make sure
that you configure available space - 5% in the nodes.conf.
This is what we do and it works great.
Kind regards, Matt
On 2023-07-24 05:48, Shunran Zhang wrote:
Hi all,
I am attempting to setup a gres to manage jobs that need a
scratch space, but only a few of our computational nodes are
equipped with SSD for such scratch space. Originally I setup a new
partition for those IO-bound jobs, but it ended up that those jobs
might be allocated to the same node thus fighting each other for
IO.
With a look over other settings it appears that the gres setting
looks promising. However I was having some difficulties figuring
out how to limit access to such space to those who requested
--gres=ssd:1.
For now I am using Flags=CountOnly to trust users who uses SSD
request for it, but apparently any job submitted to a node with
SSD can just use such space. Our scratch space implementation is 2
disks (sda and sdb) formatted to btrfs and RAID 0. What should I
do to enforce such limit on which job can use such space?
Related configurations for ref:
gres.conf: NodeName=scratch-1 Name=ssd Flags=CountOnly cgroup.conf:
ConstrainDevices=yes slurm.conf: GresTypes=gpu,ssd
NodeName=scratch-1 CPUs=88 Sockets=2 CoresPerSocket=22
ThreadsPerCore=2 RealMemory=180000 Gres=ssd:1 State=UNKNOWN
Sincerely,
S. Zhang
