I think that generally you should set this partition to drain state. This will prevent new jobs submission, but allow all running and pending jobs execution.
I don't think it's possible to change the partition for running jobs, for pending you could have done simply: scontrol update job=JOBID partition=NEWPART If you remove the partition from configuration, then in job structure you have a reference to non-existent partition. Such a structure can not be created when you restart slurm, so jobs were removed. cheers, Marcin 2017-02-13 18:12 GMT+01:00 Nicholas McCollum <[email protected]>: > Just ran into an issue while removing old nodes that I had in a > partition. We use a job_submit.lua script that has every job submitted > to every partition. > > local submit_part = "" > if asc_job_partition == "" then > asc_job_partition = "dmc,uv" > end > if string.find( asc_job_partition , 'dmc' ) then > submit_part = (submit_part .. "dmc-ivy-bridge,dmc-haswell,dmc- > broadwell,gpu_kepler,gpu_pascal,") > end > if string.find( asc_job_partition , 'uv' ) then > submit_part = (submit_part .. "uv,") > end > if string.find( asc_job_partition , 'knl' ) then > submit_part = (submit_part .. "knl,") > end > --[[ Strips the last comma off the string and writes the new info ]]- > - > job_desc.partition = string.sub(submit_part, 1, -2) > > I had previously gpu_fermi and gpu_tesla in my string. All currently > running jobs had submit partitions with this string in it, although all > of the nodes were drained in the gpu_fermi and gpu_tesla partitions. > > I went ahead and removed the line from my job_submit.lua, and removed > the partition and nodes from the slurm.conf. I then initiated a > scontrol reconfigure. > > After a minute or two I noticed then, that all the jobs had > disappeared. Nothing pending, nothing running. > > The logs showed 'error: Invalid partition (gpu_fermi) for job 70496' > and that SLURM had sent a terminate command to all of the running jobs > because of this. > > So, a cautionary tale for all. I think in the future I will edit my > job_submit.lua script and wait for all the jobs that have ran through > it to finish before removing partitions. > > My question for the group is, other than the above mentioned method, is > there something I could have done differently to prevent SLURM from > killing jobs when removing partitions? > > Thanks! > > -- > Nicholas McCollum > HPC Systems Administrator > Alabama Supercomputer Authority
