Yes we have separate partitions. Some are specific to gpu having 2 nodes with 8 gpu and another partitions are mix of both,nodes with 2 gpu and very few nodes are without any gpu.
Regards Navin On Sat, Jun 13, 2020, 21:11 navin srivastava <navin.alt...@gmail.com> wrote: > Thanks Renfro. > > Yes we have both types of nodes with gpu and nongpu. > Also some users job require gpu and some applications use only CPU. > > So the issue happens when user priority is high and waiting for gpu > resources which is not available and the job with lower priority is waiting > even though enough CPU is available which need only CPU resources. > > When I hold gpu jobs the cpu jobs will go through. > > Regards > Navin > > On Sat, Jun 13, 2020, 20:37 Renfro, Michael <ren...@tntech.edu> wrote: > >> Will probably need more information to find a solution. >> >> To start, do you have separate partitions for GPU and non-GPU jobs? Do >> you have nodes without GPUs? >> >> On Jun 13, 2020, at 12:28 AM, navin srivastava <navin.alt...@gmail.com> >> wrote: >> >> Hi All, >> >> In our environment we have GPU. so what i found is if the user having >> high priority and his job is in queue and waiting for the GPU resources >> which are almost full and not available. so the other user submitted the >> job which does not require the GPU resources are in queue even though lots >> of cpu resources are available. >> >> our scheduling mechanism is FIFO and Fair tree enabled. Is there any way >> we can make some changes so that the cpu based job should go through and >> GPU based job can wait till the GPU resources are free. >> >> Regards >> Navin. >> >> >> >> >>
