Hi all, After working with the developers of DMTCP checkpoint library, we have a nice working version of Slurm+DMTCP. We are able to checkpoint any batch job (well, most of them) and restarting it anywhere else in the cluster. We are testing it thoroughly, and will let you know in a few weeks in case any of you is interested in testing/using it.
Anyway, after having this ready, we are working on some uses to this new functionality. An interesting one is job preemption: if a job is running but another with a higher priority comes, then checkpoint the first one, cancel it, run the second, and restart the first one somewhere else. This already counts with Slurm support, so it is kind of trivial from a technical point of view. I am however not fully convinced on how this should work. If possible, I'd like to have your thoughts as expert system administrators. A key thing is that, while we are able to checkpoint/restart most of the jobs (both serial and MPI) we are able to checkpoint only batch jobs: no srun, no salloc. Also, jobs running on GPUs or Xeon Phi cannot currently be checkpointed. My question is, what should happen to this non-checkpointable jobs whenever one with higher priority comes? One alternative should be to preempt only jobs with checkpoint support, so no computation is lost; the other would be to preempt whatever necessary to run the job as soon as possible, not caring about being able to restore it later. I can imagine scenarios where one of the alternatives is better than the other one, but I am not sure of how realistic they are. As system administrators, would you have any preference on this? Also, the next question is what happens with the job to be restarted. With current Slurm implementation it goes back to the queue. The problem it this is that, if there are many jobs in the queue, this partially-completed one will have to wait a lot before restarting. From my point of view it would make sense to put it on top of the queue, so it restarts as soon as there is a free slot. This can be easily changed in the code, but I'd love to hear your point of view before modifying anything. So, any ideas/suggestions? Thanks for your help. Best regards, Manuel
