Robbert, All, Thank you for you reply.
The gres man page doesn't really make it abundantly clear to us that the gres allocation is for the job as a "whole" vs. the job's gres allocation per task. Though we do see now where states it is an allocation per node. We are also observing your point with Test #2 & #3 from my original post. However, we assumed the --gres behavior would be like "-c, --cpus-per-task=". In other words, it appears the --gres=gpu:1 is not really --gres_per_task=gpu:1, but rather --gres_per_node=gpu:1 Correct? We are surprised that slurm doesn't handle allocating a gpu_per_task already, and that we appear to be forced to use wrapper scripts. Are we in the wrong here? Thank you! On Tue, Sep 5, 2017 at 5:11 PM, Robbert Eggermont <[email protected]> wrote: > > Given: >> >> % salloc -n 4 -c 2 -gres=gpu:1 >> >> % srun env | grep CUDA # a single srun >> >> # Currently always produces >> >> CUDA_VISIBLE_DEVICES=0 >> >> CUDA_VISIBLE_DEVICES=0 >> >> CUDA_VISIBLE_DEVICES=0 >> >> CUDA_VISIBLE_DEVICES=0 >> > > man salloc: > --gres > ... The specified resources will be allocated to the job on each node. > ... > > Your requested a single GPU for the whole job; to allocate a separate GPU > for every task, you want --gres=gpu:4. > > That being said, I don't know of a way to make srun (no experience with > mpirun) exclusively assign parts of the gres to each task. > > You can do something like this (in a Bash wrapper) to exclusively assign a > GPU to a task (based on what is allocated to your job to be safe): > IFS=, cuda_devices=( $CUDA_VISIBLE_DEVICES ) > [ $SLURM_LOCALID -lt ${#cuda_devices[*]} ] || exit 1 > CUDA_VISIBLE_DEVICES=${cuda_devices[$SLURM_LOCALID]} > > Alternatively, you may be able to use arrays? > sbatch -a 0-4 -c 2 --gres=gpu:1 <command> > > Best, > > Robbert >
