Hi Jürgen,This was it! Thank you so much for the hint! I did not know about the "top" command and was also not aware that this option was enabled in our slurm.conf.
Thanks for the help! Sebastian On 03.05.23 12:10, Juergen Salk wrote:
Hi Sebastian, maybe it's a silly thought on my part, but do you have the `enable_user_top´ Option included in your SchedulerParameters configuration? This would allow regular users to use `scontrol top <job_list>´ to push some of their jobs ahead of other jobs owned by them and this works internally by adjusting the nice values of the specified jobs. I may be totally wrong, but if I remember correctly it is not recommended to configure SchedulerParameters=enable_user_top in general, though, because regular user use of `scontrol top´ is (or was?) supposed to introduce bad side effects in certain scenarios that would allow users to push pending jobs ahead of normal (also other user's) jobs in the queue, if only one of their jobs has already a negative nice value assigned, e.g. by an administrator. Best regards Jürgen * Sebastian Potthoff <s.potth...@uni-muenster.de> [230503 10:36]:Hello all, I am encountering some unexpected behavior where the jobs (queued & running) of one specific user have negative NICE values and therefore an increased priority. The user is not privileged in any way and cannot explicitly set the nice value to a negative value by e.g. adding "--nice=-INT" . There are also no QoS which would allow this (is this even possible?). The cluster is using the "priority/multifactor" plugin with weights set for Age, FaireShare and JobSize. This is the only user on the whole cluster where this occurs. From what I can tell, he/she is not doing anything out of the ordinary. However, in the job scripts the user does set a nice value of "0". The user also uses some "strategy" where he/she submits the same job to multiple partitions and, as soon as one of these jobs starts, all other jobs (with the same jobname) will be set on "hold". Does anyone have an idea how this could happen? Does Slurm internally adjust the NICE values in certain situations? (I searched the sources but couldn't find anything that would suggest this). Slurm version is 23.02.1 Example squeue output: [root@mgmt ~]# squeue -u USERID -O JobID,Nice JOBID NICE 14846760 -5202 14846766 -8988 14913146 -13758 14917361 -15103 Any hints are appreciated. Kind regards Sebastian
-- Westfälische Wilhelms-Universität (WWU) Münster WWU IT Sebastian Potthoff, M.Sc. (eScience/HPC) Röntgenstraße 7-13, R.207/208 48149 Münster Tel. +49 251 83-31640 E-Mail: s.potth...@uni-muenster.de Website: www.uni-muenster.de/it
smime.p7s
Description: S/MIME Cryptographic Signature
