[slurm-users] How to apply for multiple GPU cards from different worker nodes?

[Ran Du]

Dear all,

     Does anyone know how to set #SBATCH options to get multiple GPU cards
from different worker nodes?

     One of our users would like to apply for 16 NVIDIA V100 cards for his
job, and  there are 8 GPU cards on each worker node, I have tried the
following #SBATCH options:

      #SBATCH --partition=gpu
      #SBATCH --qos=normal
      #SBATCH --account=u07
      #SBATCH --job-name=cross
      #SBATCH --nodes=2
      #SBATCH --mem-per-cpu=1024
      #SBATCH --output=test.32^4.16gpu.log
      #SBATCH --gres=gpu:v100:16

      but got the sbatch error message :
      sbatch: error: Batch job submission failed: Requested node
configuration is not available

      And I found a similar question on stack overflow:

https://stackoverflow.com/questions/45200926/how-to-access-to-gpus-on-different-nodes-in-a-cluster-with-slurm

      And it is said that multiple GPU cards allocation on different worker
nodes are not available, the post is in 2017, is it still true at present?

      Thanks a lot for your help.

Best regards,
Ran