The more flexible way to do this is with QoS. (PreemptType=preempt/qos) You'll need to have Accounting enabled and you'll probably want qos listed in AccountingStorageEnforce. Once you do that you create a "shared" for the scavenger jobs, a QoS for each group that buys into resources. Assign the correct number of resources to each group's QoS as GrpTRES limits, add the "shared" qos as preemptable from that qos and give the qos a higher priority.
Mike Robbert
On 4/10/20, 15:34, "slurm-users on behalf of Renata Maria Dart"
<slurm-users-boun...@lists.schedmd.com on behalf of ren...@slac.stanford.edu>
wrote:
CAUTION: This email originated from outside of the Colorado School of Mines
organization. Do not click on links or open attachments unless you recognize
the sender and know the content is safe.
Hi, we have 40 nodes (all the same, amd nodes with 128 cores) which
have all been purchased by different groups at our lab and each group
would like to have immediate access of course to what they have paid
for. The stakeholder groups are also fine with allowing the general
public to use their hosts/cores provided they can preempt the general
public's jobs. One way I can see to do that is to assign specific
nodes to each stakeholder group defined as a partition, something like this:
PartitionName=shared Default=yes Priority=10 MaxTime=5-00:00:00
DefaultTime=30 PreemptMode=CANCEL State=UP Nodes=amd[0001-0040]
PartitionName=exp1 Default=no Priority=50 MaxTime=5-00:00:00
DefaultTime=1-00:00:00 PreemptMode=OFF State=UP Nodes=amd[0001-0003]
PartitionName=exp2 Default=no Priority=50 MaxTime=5-00:00:00
DefaultTime=1-00:00:00 PreemptMode=OFF State=UP Nodes=amd[0004-0019]
PartitionName=exp3 Default=no Priority=50 MaxTime=5-00:00:00
DefaultTime=1-00:00:00 PreemptMode=OFF State=UP Nodes=amd[0020-0040]
Is this the most efficient and best use of resources? In the above
scenario if scavenger jobs are running on a given experiment's hosts
and the experiment needs to run jobs, then scavenger jobs get
preempted, even if there are idle hosts in the other stakeholder
partitions. Is there a way to guarantee say exp1 that they will have
priority on 386 cores but not necessarily tie them to 3 specific hosts?
Thanks,
Renata
smime.p7s
Description: S/MIME cryptographic signature
