I'm in the exact same situation. We just made the switch from torque/maui to slurm a couple weeks ago and this was one of the few things I was not able to replicate well using slurm. Our torque/maui setup was like this:
create queue debug set queue debug queue_type = Execution set queue debug resources_max.nodect = 8 set queue debug resources_max.walltime = 01:00:00 set queue debug resources_min.ncpus = 1 set queue debug resources_min.nodect = 1 set queue debug resources_default.ncpus = 1 set queue debug resources_default.nodect = 1 set queue debug resources_default.walltime = 01:00:00 set queue debug enabled = True set queue debug started = True create queue normal set queue normal queue_type = Execution set queue normal resources_max.walltime = 08:00:00 set queue normal resources_default.walltime = 04:00:00 set queue normal enabled = True set queue normal started = True And the maui standing reservation would grab 8 nodes during the day with: SRCFG[debug] STARTTIME=8:00:00 ENDTIME=17:00:00 SRCFG[debug] DAYS=Mon,Tue,Wed,Thu,Fri SRCFG[debug] CLASSLIST=debug SRCFG[debug] TASKCOUNT=8 SRCFG[debug] HOSTLIST=r1i0n[0-7]$ Under slurm I tried to define all nodes in a partition called "normal" and a small subset of those nodes in a partition called "debug". PartitionName=normal Nodes=r[1-5]i[0-3]n[0-15] DefaultTime=04:00:00 MaxTime=08:00:00 State=UP Default=YES PartitionName=debug Nodes=r1i0n[0-7] DefaultTime=01:00:00 MaxTime=01:00:00 State=UP MaxNodes=6 Then to reserve the debug nodes during the day: scontrol create reservation ReservationName=daily_debug account=aerolab \ PartitionName=debug starttime=8:00:00 duration=9:00:00 flags=daily,part_nodes The nice thing under maui was that all you had to do was submit to the debug queue (partition). If it was during normal working hours on a weekday, the debug jobs would run on the reserved nodes. Otherwise, debug jobs (say on a weekend) would effectively run in the normal queue like any other job. Under slurm you have to explicitly request to use the debug reservation when it is in effect (in addition to the debug partition): #SBATCH -p debug #SBATCH --reservation=daily_debug The problem with the above setup is that slurm is quite literal about the reservation request. By this I mean that jobs submitted to "-p debug" at night (i.e. after 5 pm) won't run because the "daily_debug" reservation isn't in effect – the job wouldn't run until the next morning. And the converse is true as well – jobs submitted to "-p debug" (without –reservation=daily_debug) during the day won't run because it doesn't explicitly ask to use the reservation. So users would have to adjust the job settings depending on when the job was submitted – not ideal. In the end, I just took the debug nodes out of the normal partition. In other words, we have debug nodes 24/7. This was the simplest thing to do to avoid the redefinition of partitions via cron as Gary suggested. Our cluster has grown quite a bit since we first set up this debug standing reservation so having dedicated debug nodes isn't as big of a deal for us now. But if there is an elegant way to accomplish the same setup under slurm, I would appreciate knowing how to do that. Darby -----Original Message----- From: "Skouson, Gary B" <[email protected]> Reply-To: slurm-dev <[email protected]> Date: Friday, January 27, 2017 at 11:06 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] Daytime Interactive jobs We'd like to have some nodes available during the workday exclusively for interactive or debug jobs. These are fairly small, short running jobs. I'd like to make these nodes available for other jobs at night and on weekends. When using Moab, we used the standing reservation stuff to configure a reservation on for specific nodes and only allowed these kinds of jobs. I can't see an elegant way to do something like that in Slurm. I can get close to the same functionality, if I make a reservation for each weekday morning and then use a cron job to reconfigure partitions. I'll need another cron job to reconfigure the partitions again in the evening. It seems like there should be a better way to do this. Has anyone else done something like this? ----- Gary Skouson
