Here is another example. Below is a snapshot of all the jobs in our queue
right now. Job 2796 is the highest priority job that isn't already running and
this appears to be preventing any other jobs that has any overlapping features
from running. We have 72 has nodes and 72 bro nodes - so in this state 12 free
has nodes and 48 free bro nodes. Jobs 2798, 2799, 2804 and 2803 should all
start since there are no other jobs requesting bro nodes. I increased the
logging verbosity a little but I don't see much more useful info there.
It seems like others may have some plugins/job filters for solving this problem
- each processor/node type gets its own partition and a plugin or filter
submits to all partitions. If someone has some more info on this kind of
setup, I would appreciate more details.
Thanks,
Darby
Job ID Username Queue Jobname N:ppn Proc Wall S Elap Prio
Reason Features
-------- -------- ------- -------------------- ----- ---- ----- - ----- ------
---------- --------------
2781 bstewart normal DJ1 30:24 720 08:00 R 06:49 11244
None has
2789 pjang normal Run10 30:24 720 08:00 R 00:27 11381
None has
2796 bstewart normal SIN 30:24 720 08:00 Q 00:00 11230
Resources has
2797 flumpkin normal DJ3 24:24 576 08:00 R 06:47 10956
None bro
2798 ema normal m0.90a120_w3_dragon+ 4:24 96 08:00 Q 00:00 11175
Priority [has|bro]
2799 ema normal m0.90a120r-90_w3_dra 4:24 96 08:00 Q 00:00 11172
Priority [has|bro]
2800 dgs normal m0.251874a0.0_crmdcf 26:24 624 08:00 Q 00:00 11030
Priority [has]
2801 flumpkin normal DJ2 36:16 576 08:00 R 01:38 10956
None san
2802 dgs normal m0.0a0_proctor 16:12 192 08:00 R 01:34 10985
None [wes]
2803 dvicker normal slurm_test 4:24 96 01:00 Q 00:00 11206
Priority [has|bro]
2804 dvicker normal slurm_test 4:24 96 01:00 Q 00:00 11206
Priority [bro|has]
% sprio
JOBID PRIORITY AGE FAIRSHARE JOBSIZE PARTITION
2796 11230 284 0 947 10000
2798 11175 148 36 992 10000
2799 11172 144 36 992 10000
2800 11030 107 1 923 10000
2803 11206 0 215 992 10000
2804 11206 0 215 992 10000
[2017-02-15T08:12:14.395] debug: Spawning ping agent for
r1i3n[2-15],r2i0n[0-15],r2i1n[0-15],r2i2n[0-3],r4i3n[6-17],r4i4n[0-17],r4i5n[0-17],r4i6n[0-1],r5i0n[16-17],r5i1n[0-17],r5i2n[0-17],r5i3n[0-11]
[2017-02-15T08:12:14.395] debug: Spawning registration agent for
r3i1n[6-15],r3i2n[0-15],r3i3n[0-13],r4i0n[0-9] 50 hosts
[2017-02-15T08:12:17.397] debug: sched: Running job scheduler
[2017-02-15T08:12:17.397] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:12:17.397] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:12:17.397] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:12:22.066] debug: Job 2803 feature list: [has|bro]
[2017-02-15T08:12:22.069] _slurm_rpc_submit_batch_job JobId=2803 usec=3524
[2017-02-15T08:12:22.401] debug: sched: Running job scheduler
[2017-02-15T08:12:22.401] debug: sched: JobId=2803. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:12:22.401] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:12:22.401] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:12:22.401] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:12:32.223] debug: backfill: beginning
[2017-02-15T08:12:32.223] debug: backfill: 5 jobs to backfill
[2017-02-15T08:12:46.062] debug: Job 2804 feature list: [bro|has]
[2017-02-15T08:12:46.066] _slurm_rpc_submit_batch_job JobId=2804 usec=4438
[2017-02-15T08:12:46.415] debug: sched: Running job scheduler
[2017-02-15T08:12:46.415] debug: sched: JobId=2803. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:12:46.415] debug: sched: JobId=2804. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:12:46.415] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:12:46.415] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:12:46.415] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:13:02.224] debug: backfill: beginning
[2017-02-15T08:13:02.224] debug: backfill: 6 jobs to backfill
[2017-02-15T08:13:07.428] debug: Updating partition uid access list
[2017-02-15T08:13:17.434] debug: sched: Running job scheduler
[2017-02-15T08:13:17.434] debug: sched: JobId=2803. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:13:17.434] debug: sched: JobId=2804. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:13:17.434] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:13:17.434] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:13:17.434] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:13:32.225] debug: backfill: beginning
[2017-02-15T08:13:32.225] debug: backfill: 6 jobs to backfill
[2017-02-15T08:13:54.471] debug: Spawning ping agent for
r1i0n[0-15],r1i1n[0-15],r1i2n[0-15],r1i3n[0-1],r2i2n[4-15],r2i3n[0-15],r3i0n[0-15],r3i1n[0-5],r4i6n[2-17],r4i7n[0-17],r5i0n[0-15],r5i3n[12-17]
[2017-02-15T08:13:54.471] debug: Spawning registration agent for
r4i0n[10-17],r4i1n[0-17],r4i2n[0-17],r4i3n[0-5] 50 hosts
[2017-02-15T08:14:02.227] debug: backfill: beginning
[2017-02-15T08:14:02.227] debug: backfill: 6 jobs to backfill
[2017-02-15T08:14:17.485] debug: sched: Running job scheduler
[2017-02-15T08:14:17.485] debug: sched: JobId=2803. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:14:17.485] debug: sched: JobId=2804. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:14:17.485] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:14:17.485] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:14:17.485] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:14:32.228] debug: backfill: beginning
[2017-02-15T08:14:32.228] debug: backfill: 6 jobs to backfill
[2017-02-15T08:15:17.534] debug: sched: Running job scheduler
[2017-02-15T08:15:17.534] debug: sched: JobId=2803. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:15:17.534] debug: sched: JobId=2804. State=PENDING.
Reason=Priority, Priority=11206. Partition=normal.
[2017-02-15T08:15:17.534] debug: sched: JobId=2798. State=PENDING.
Reason=Priority, Priority=11175. Partition=normal.
[2017-02-15T08:15:17.534] debug: sched: JobId=2799. State=PENDING.
Reason=Priority, Priority=11172. Partition=normal.
[2017-02-15T08:15:17.534] debug: sched: JobId=2800. State=PENDING.
Reason=Priority, Priority=11030. Partition=normal.
[2017-02-15T08:15:18.230] debug: backfill: beginning
[2017-02-15T08:15:18.230] debug: backfill: 6 jobs to backfill
On Feb 13, 2017, at 9:13 AM, Vicker, Darby (JSC-EG311)
<[email protected]<mailto:[email protected]>> wrote:
BTW, I should have mentioned we are using 16.05.7.
Thanks for the info Chris. I don't think this is a backfill problem. These
are jobs that should start regardless of backfill – they are top priority jobs
that have free resources available. We are using the default backfill options
but those should be fine for us. The default backfill queue depth is 100 jobs
and we normally have less than 100 total jobs in the queue at a given time.
Can you point me to some info on the plugins and filters you mentioned? I'm
having trouble finding info on them.
-----Original Message-----
From: Christopher Samuel <[email protected]<mailto:[email protected]>>
Reply-To: slurm-dev <[email protected]<mailto:[email protected]>>
Date: Wednesday, February 8, 2017 at 5:32 PM
To: slurm-dev <[email protected]<mailto:[email protected]>>
Subject: [slurm-dev] Re: Job priority/cluster utilization help
On 08/02/17 11:19, Vicker, Darby (JSC-EG311) wrote:
Sorry for the long post but not sure how to get adequate help without
providing a lot of detail. Any recommendations on configuring the
scheduler to help these jobs run and increase the cluster utilization
would be appreciated.
My one thought after a quick scan is that both the jobs you mention are
listed as reason "Priority" and there's a higher priority job 1772 in
the list before them. You might want to look at your backfill settings
to see whether it's looking far enough down the queue to see these.
Perhaps an alternative idea would be to instead of using features use
partitions and then have people submit to all partitions (there is a
plugin for that, though we use a submit filter instead to accomplish the
same).
That way Slurm should consider each job against each partition (set of
architectures) individually.
Best of luck!
Chris
--
Christopher Samuel Senior Systems Administrator
VLSCI - Victorian Life Sciences Computation Initiative
Email: [email protected]<mailto:[email protected]> Phone: +61 (0)3 903
55545
http://www.vlsci.org.au/ http://twitter.com/vlsci
