Hi everyone,
There was a post along these lines not long ago but slightly different and no
real solution so I'm starting a new thread. We have a fairly old cluster with
4 generations of processor types. We mostly run multi-core MPI jobs so need to
have consistent processor types for a given job. We recently switched from
torque/maui to slurm and one of the big motivations for this was to utilize the
ability to specify the total core count and let slurm pick the processor type
(as opposed to the nodes=X:ppn=Y:feature model in torque/maui). This is
accomplished by putting all nodes in the same partition, assigning features to
the different node types and using this in the batch script:
#SBATCH -n 288
#SBATCH --constraint=[wes|san|has|bro]
This often works great – as the cluster gets loaded up the jobs fill in where
there are available nodes. But other times jobs seem to get stuck in a pending
state that should run – i.e. there are free nodes in which the jobs could run.
In the example below, jobs 1802 and 1830 are both eligible to run on any
processor type and there are enough free san and has nodes for either of those
jobs to run. These will stay in a queued state for hours if we leave them.
But if we kill those jobs and change the constraint list to have only one
processor type (with enough free nodes at the time), the jobs start right away.
Sorry for the long post but not sure how to get adequate help without providing
a lot of detail. Any recommendations on configuring the scheduler to help
these jobs run and increase the cluster utilization would be appreciated.
Thanks,
Darby
Job ID Username Queue Jobname N:ppn Proc Wall S Elap
Reason Features
-------- -------- ------- -------------------- ----- ---- ----- - -----
---------- --------------
1754 gsalaza3 normal cat_tile 12:12 144 08:00 R 05:01
None [wes]
1755 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:59
None [wes]
1756 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:56
None [wes]
1757 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:55
None [wes]
1758 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:51
None [wes]
1759 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:51
None [wes]
1760 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:48
None [wes]
1761 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:48
None [wes]
1765 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:43
None [wes]
1766 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:24
None [wes]
1767 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:24
None [wes]
1768 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:13
None [wes]
1769 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:13
None [wes]
1770 gsalaza3 normal cat_tile 12:12 144 08:00 R 04:11
None [wes]
1771 gsalaza3 normal cat_tile 12:12 144 08:00 R 01:19
None [wes]
1772 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Resources [wes]
1773 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1774 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1775 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1776 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1778 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1779 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1780 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1781 gsalaza3 normal cat_tile 12:12 144 08:00 Q 00:00
Priority [wes]
1782 bstewart normal N2O2 25:12 300 08:00 Q 00:00
Priority wes
1785 dgs normal m0.251874a0.0_crmdcf 24:12 288 08:00 Q 00:00
Priority [wes]
1788 bstewart normal DJ1 30:24 720 08:00 R 03:55
None has
1802 benkirk normal snc-e10-e15 7:23 164 04:00 Q 00:00
Priority [wes|san|has|bro]
1805 flumpkin normal DJ3 24:24 576 08:00 R 03:08
None bro
1811 jwillems normal Orion_For_Gio-222 6:16 96 08:00 R 02:51
None [san]
1824 jwillems normal Orion_For_Gio-111 6:16 96 08:00 R 01:32
None [san]
1827 bstewart normal SIN 30:24 720 08:00 R 01:11
None has
1828 bstewart normal QUAD 30:24 720 08:00 R 01:05
None bro
1829 pmcclou1 normal Run_22_SST 18:16 288 08:00 R 00:51
None [san]
1830 larin normal SpXFB10_perfAir_100 9:16 144 08:00 Q 00:00
Priority [wes|san|has|bro]
1831 jwillems normal Boeing_Pretest-221 6:16 96 08:00 R 00:25
None [san]
Stats:
total (408/6912) || ppn=12 (192/2304) | ppn=16 ( 72/1152) | ppn=24 (144/3456)
S Node CPU Job || S Node CPU Job | S Node CPU Job | S Node CPU Job
- ---- ----- ---- || - ---- ----- ---- | - ---- ----- ---- | - ---- ----- ----
R 330 5472 23 || R 180 2160 15 | R 36 576 4 | R 114 2736 4
- ---- ----- ---- || - ---- ----- ---- | - ---- ----- ---- | - ---- ----- ----
R 80% 79% || R 93% 93% | R 50% 50% | R 79% 79%
- ---- ----- ---- || - ---- ----- ---- | - ---- ----- ---- | - ---- ----- ----
F 78 1440 || F 12 144 | F 36 576 | F 30 720
# scontrol show job 1802
JobId=1802 JobName=snc-e10-e15
UserId=benkirk(3170) GroupId=eg3(3000) MCS_label=N/A
Priority=11244 Nice=0 Account=aerolab QOS=normal
JobState=PENDING Reason=Priority Dependency=(null)
Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=04:00:00 TimeMin=N/A
SubmitTime=2017-02-07T13:25:06 EligibleTime=2017-02-07T13:25:06
StartTime=Unknown EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=normal AllocNode:Sid=service1:10980
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null)
NumNodes=7 NumCPUs=164 NumTasks=164 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=164,node=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=[wes|san|has|bro] Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
# scontrol show job 1830
JobId=1830 JobName=SpXFB10_perfAir_100
UserId=larin(3188) GroupId=eg3(3000) MCS_label=N/A
Priority=11258 Nice=0 Account=aerolab QOS=normal
JobState=PENDING Reason=Priority Dependency=(null)
Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=08:00:00 TimeMin=N/A
SubmitTime=2017-02-07T16:13:25 EligibleTime=2017-02-07T16:13:25
StartTime=Unknown EndTime=Unknown Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=normal AllocNode:Sid=service1:8601
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null)
NumNodes=9 NumCPUs=144 NumTasks=144 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=144,node=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=[wes|san|has|bro] Gres=(null) Reservation=(null)
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
# date
Tue Feb 7 18:05:26 CST 201
# sprio -l
JOBID USER PRIORITY AGE FAIRSHARE JOBSIZE PARTITION
QOS NICE TRES
1772 gsalaza3 11258 193 78 988 10000
0 0
1773 gsalaza3 11256 191 78 988 10000
0 0
1774 gsalaza3 11256 191 78 988 10000
0 0
1775 gsalaza3 11256 190 78 988 10000
0 0
1776 gsalaza3 11256 190 78 988 10000
0 0
1778 gsalaza3 11254 188 78 988 10000
0 0
1779 gsalaza3 11253 188 78 988 10000
0 0
1780 gsalaza3 11253 188 78 988 10000
0 0
1781 gsalaza3 11253 188 78 988 10000
0 0
1782 bstewart 11161 185 0 977 10000
0 0
1785 dgs 11160 178 5 978 10000
0 0
1802 benkirk 11188 137 65 987 10000
0 0
1830 larin 11202 20 194 988 10000
0 0
slurmctl log:
[2017-02-07T13:14:28.678] job_complete: JobID=1786 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T13:14:28.680] job_complete: JobID=1786 State=0x8003 NodeCnt=6 done
[2017-02-07T13:16:22.423] update_node: node r4i0n6 state set to IDLE
[2017-02-07T13:19:40.882] _slurm_rpc_submit_batch_job JobId=1798 usec=8693
[2017-02-07T13:19:53.255] backfill: Started JobId=1798 in normal on
r4i1n[14-17],r4i2n[0-17],r4i3n[0-7]
[2017-02-07T13:19:56.255] _pick_step_nodes: Configuration for job 1798 is
complete
[2017-02-07T13:20:23.721] _slurm_rpc_submit_batch_job JobId=1799 usec=3960
[2017-02-07T13:23:01.563] _slurm_rpc_submit_batch_job JobId=1800 usec=8934
[2017-02-07T13:23:28.457] backfill: Started JobId=1800 in normal on r4i3n[8-13]
[2017-02-07T13:23:28.660] _pick_step_nodes: Configuration for job 1800 is
complete
[2017-02-07T13:24:15.226] _slurm_rpc_submit_batch_job JobId=1801 usec=3598
[2017-02-07T13:24:28.524] backfill: Started JobId=1801 in normal on
r4i0n[3-6],r4i3n[14-15]
[2017-02-07T13:24:29.102] _pick_step_nodes: Configuration for job 1801 is
complete
[2017-02-07T13:25:06.961] _slurm_rpc_submit_batch_job JobId=1802 usec=8405
[2017-02-07T13:26:07.603] job_complete: JobID=1798 State=0x1 NodeCnt=30
WEXITSTATUS 0
[2017-02-07T13:26:07.606] job_complete: JobID=1798 State=0x8003 NodeCnt=30 done
[2017-02-07T13:26:28.036] job_complete: JobID=1801 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T13:26:37.451] _slurm_rpc_submit_batch_job JobId=1803 usec=6168
[2017-02-07T13:26:58.661] backfill: Started JobId=1803 in normal on
r4i1n[14-17],r4i2n[0-1]
[2017-02-07T13:26:58.934] _pick_step_nodes: Configuration for job 1803 is
complete
[2017-02-07T13:27:17.922] job_complete: JobID=1803 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T13:28:49.621] _slurm_rpc_submit_batch_job JobId=1804 usec=9253
[2017-02-07T13:28:49.744] backfill: Started JobId=1804 in normal on
r4i1n[14-17],r4i2n[0-1]
[2017-02-07T13:28:49.937] _pick_step_nodes: Configuration for job 1804 is
complete
[2017-02-07T13:29:46.073] job_complete: JobID=1787 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T13:29:46.075] job_complete: JobID=1787 State=0x8003 NodeCnt=6 done
[2017-02-07T13:31:03.621] _slurm_rpc_submit_batch_job JobId=1805 usec=4114
[2017-02-07T13:31:03.771] job_complete: JobID=1716 State=0x1 NodeCnt=24
WEXITSTATUS 0
[2017-02-07T13:31:03.773] job_complete: JobID=1716 State=0x8003 NodeCnt=24 done
[2017-02-07T13:31:19.889] backfill: Started JobId=1805 in normal on
r5i1n[12-17],r5i2n[0-17]
[2017-02-07T13:31:21.830] _pick_step_nodes: Configuration for job 1805 is
complete
[2017-02-07T13:33:26.507] _slurm_rpc_submit_batch_job JobId=1806 usec=6201
[2017-02-07T13:33:49.608] _slurm_rpc_submit_batch_job JobId=1807 usec=3983
[2017-02-07T13:33:58.039] backfill: Started JobId=1807 in normal on
r4i0n[14-17],r4i1n[0-1],r4i2n[2-17],r4i3n[0-7]
[2017-02-07T13:34:02.633] _pick_step_nodes: Configuration for job 1807 is
complete
[2017-02-07T13:36:58.981] _slurm_rpc_submit_batch_job JobId=1808 usec=3878
[2017-02-07T13:37:57.985] _slurm_rpc_submit_batch_job JobId=1809 usec=4259
[2017-02-07T13:37:58.228] backfill: Started JobId=1809 in normal on
r4i0n[3-6],r4i3n[14-15]
[2017-02-07T13:37:58.428] _pick_step_nodes: Configuration for job 1809 is
complete
[2017-02-07T13:41:46.630] job_complete: JobID=1748 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T13:42:41.848] job_complete: JobID=1793 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T13:44:13.501] _slurm_rpc_submit_batch_job JobId=1810 usec=10330
[2017-02-07T13:48:20.368] _slurm_rpc_submit_batch_job JobId=1811 usec=4507
[2017-02-07T13:48:28.700] backfill: Started JobId=1811 in normal on r4i0n[8-13]
[2017-02-07T13:48:28.925] _pick_step_nodes: Configuration for job 1811 is
complete
[2017-02-07T13:48:56.808] job_complete: JobID=1807 State=0x8004 NodeCnt=30
WTERMSIG 15
[2017-02-07T14:01:38.985] _slurm_rpc_submit_batch_job JobId=1812 usec=3736
[2017-02-07T14:02:27.043] job_complete: JobID=1800 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T14:02:27.046] job_complete: JobID=1800 State=0x8003 NodeCnt=6 done
[2017-02-07T14:02:29.440] job_complete: JobID=1763 State=0x1 NodeCnt=4
WEXITSTATUS 0
[2017-02-07T14:02:29.440] email msg to [email protected]: SLURM
Job_id=1763 Name=suit_backpack Ended, Run time 02:07:34, COMPLETED, ExitCode 0
[2017-02-07T14:02:29.442] job_complete: JobID=1763 State=0x8003 NodeCnt=4 done
[2017-02-07T14:05:53.788] _slurm_rpc_submit_batch_job JobId=1813 usec=4581
[2017-02-07T14:07:55.192] job_complete: JobID=1804 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T14:08:15.263] _slurm_rpc_submit_batch_job JobId=1814 usec=4363
[2017-02-07T14:08:28.645] backfill: Started JobId=1814 in normal on
r4i0n[14-17],r4i1n[0-1]
[2017-02-07T14:08:28.843] _pick_step_nodes: Configuration for job 1814 is
complete
[2017-02-07T14:09:45.787] _slurm_rpc_submit_batch_job JobId=1815 usec=4093
[2017-02-07T14:11:44.739] job_complete: JobID=1795 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T14:11:44.741] job_complete: JobID=1795 State=0x8003 NodeCnt=6 done
[2017-02-07T14:14:18.506] job_complete: JobID=1809 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T14:14:18.508] job_complete: JobID=1809 State=0x8003 NodeCnt=6 done
[2017-02-07T14:15:21.706] _slurm_rpc_submit_batch_job JobId=1816 usec=4354
[2017-02-07T14:15:28.000] backfill: Started JobId=1816 in normal on r4i1n[2-7]
[2017-02-07T14:15:28.216] _pick_step_nodes: Configuration for job 1816 is
complete
[2017-02-07T14:15:41.155] _slurm_rpc_submit_batch_job JobId=1817 usec=4180
[2017-02-07T14:17:35.980] _slurm_rpc_submit_batch_job JobId=1818 usec=4220
[2017-02-07T14:17:58.125] backfill: Started JobId=1818 in normal on r4i1n[8-13]
[2017-02-07T14:17:58.860] _pick_step_nodes: Configuration for job 1818 is
complete
[2017-02-07T14:18:19.267] job_complete: JobID=1818 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T14:19:01.952] _slurm_rpc_submit_batch_job JobId=1819 usec=4544
[2017-02-07T14:19:28.203] backfill: Started JobId=1819 in normal on r4i1n[8-13]
[2017-02-07T14:19:28.741] _pick_step_nodes: Configuration for job 1819 is
complete
[2017-02-07T14:27:35.200] job_complete: JobID=1819 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T14:27:42.759] _slurm_rpc_submit_batch_job JobId=1820 usec=3842
[2017-02-07T14:27:58.566] backfill: Started JobId=1820 in normal on r4i1n[8-13]
[2017-02-07T14:27:59.182] _pick_step_nodes: Configuration for job 1820 is
complete
[2017-02-07T14:29:09.829] job_complete: JobID=1814 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T14:30:25.300] _slurm_rpc_submit_batch_job JobId=1821 usec=4228
[2017-02-07T14:30:28.691] backfill: Started JobId=1821 in normal on
r4i0n[14-17],r4i1n[0-1]
[2017-02-07T14:30:28.879] _pick_step_nodes: Configuration for job 1821 is
complete
[2017-02-07T14:33:14.236] job_complete: JobID=1816 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T14:33:24.597] _slurm_rpc_submit_batch_job JobId=1822 usec=4241
[2017-02-07T14:33:27.820] backfill: Started JobId=1822 in normal on r4i1n[2-7]
[2017-02-07T14:33:27.991] _pick_step_nodes: Configuration for job 1822 is
complete
[2017-02-07T14:41:25.163] update_job: setting comment to None for job_id 1817
[2017-02-07T14:41:25.163] sched: update_job: releasing hold for job_id 1817 uid
3344
[2017-02-07T14:41:25.163] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=7123
[2017-02-07T14:48:03.488] _slurm_rpc_submit_batch_job JobId=1823 usec=3445
[2017-02-07T14:57:02.908] update_job: setting comment to None for job_id 1823
[2017-02-07T14:57:02.908] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=201
[2017-02-07T14:59:08.258] update_job: setting comment to None for job_id 1823
[2017-02-07T14:59:08.258] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=249
[2017-02-07T15:01:13.924] update_job: setting comment to None for job_id 1823
[2017-02-07T15:01:13.944] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=19554
[2017-02-07T15:03:18.959] update_job: setting comment to None for job_id 1823
[2017-02-07T15:03:18.959] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=261
[2017-02-07T15:05:24.233] update_job: setting comment to None for job_id 1823
[2017-02-07T15:05:24.233] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=203
[2017-02-07T15:06:35.032] job_complete: JobID=1821 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T15:07:08.953] _slurm_rpc_submit_batch_job JobId=1824 usec=3830
[2017-02-07T15:07:29.948] update_job: setting comment to None for job_id 1823
[2017-02-07T15:07:29.948] _slurm_rpc_update_job complete JobId=(null) uid=3344
usec=266
[2017-02-07T15:07:34.444] backfill: Started JobId=1824 in normal on
r4i0n[14-17],r4i1n[0-1]
[2017-02-07T15:07:34.627] _pick_step_nodes: Configuration for job 1824 is
complete
[2017-02-07T15:10:05.402] _slurm_rpc_submit_batch_job JobId=1825 usec=3458
[2017-02-07T15:10:17.713] job_complete: JobID=1822 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T15:10:31.005] _slurm_rpc_submit_batch_job JobId=1826 usec=4617
[2017-02-07T15:10:57.594] backfill: Started JobId=1826 in normal on r4i1n[2-7]
[2017-02-07T15:10:57.764] _pick_step_nodes: Configuration for job 1826 is
complete
[2017-02-07T15:20:51.666] job_complete: JobID=1749 State=0x1 NodeCnt=12
WEXITSTATUS 0
[2017-02-07T15:20:51.669] job_complete: JobID=1749 State=0x8003 NodeCnt=12 done
[2017-02-07T15:20:51.839] sched: Allocate JobID=1771
NodeList=r1i3n[12-15],r2i0n[0-7] #CPUs=144 Partition=normal
[2017-02-07T15:20:52.069] _pick_step_nodes: Configuration for job 1771 is
complete
[2017-02-07T15:28:18.772] _slurm_rpc_submit_batch_job JobId=1827 usec=3974
[2017-02-07T15:28:18.787] job_complete: JobID=1717 State=0x1 NodeCnt=30
WEXITSTATUS 0
[2017-02-07T15:28:18.789] job_complete: JobID=1717 State=0x8003 NodeCnt=30 done
[2017-02-07T15:28:28.348] backfill: Started JobId=1827 in normal on
r4i5n[12-17],r4i6n[0-17],r4i7n[0-5]
[2017-02-07T15:28:30.302] _pick_step_nodes: Configuration for job 1827 is
complete
[2017-02-07T15:34:35.206] _slurm_rpc_submit_batch_job JobId=1828 usec=4574
[2017-02-07T15:34:35.222] job_complete: JobID=1718 State=0x1 NodeCnt=30
WEXITSTATUS 0
[2017-02-07T15:34:35.224] job_complete: JobID=1718 State=0x8003 NodeCnt=30 done
[2017-02-07T15:34:57.585] backfill: Started JobId=1828 in normal on
r5i0n[0-17],r5i1n[0-11]
[2017-02-07T15:34:59.526] _pick_step_nodes: Configuration for job 1828 is
complete
[2017-02-07T15:48:47.984] _slurm_rpc_submit_batch_job JobId=1829 usec=4336
[2017-02-07T15:48:48.086] backfill: Started JobId=1829 in normal on
r4i1n[14-17],r4i2n[0-13]
[2017-02-07T15:48:52.496] _pick_step_nodes: Configuration for job 1829 is
complete
[2017-02-07T16:09:26.496] job_complete: JobID=1820 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T16:09:26.498] job_complete: JobID=1820 State=0x8003 NodeCnt=6 done
[2017-02-07T16:13:25.670] _slurm_rpc_submit_batch_job JobId=1830 usec=4433
[2017-02-07T16:14:10.686] _slurm_rpc_submit_batch_job JobId=1831 usec=4371
[2017-02-07T16:14:27.998] backfill: Started JobId=1831 in normal on r4i1n[8-13]
[2017-02-07T16:14:28.654] _pick_step_nodes: Configuration for job 1831 is
complete
[2017-02-07T16:20:55.107] job_complete: JobID=1826 State=0x8004 NodeCnt=6
WTERMSIG 15
[2017-02-07T16:42:39.817] job_complete: JobID=1811 State=0x1 NodeCnt=6
WEXITSTATUS 0
[2017-02-07T16:42:39.819] job_complete: JobID=1811 State=0x8003 NodeCnt=6 done
slurm.conf:
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/linear
FastSchedule=1
PriorityType=priority/multifactor
PriorityDecayHalfLife=7-0
PriorityWeightFairshare=1000
PriorityWeightAge=1000
PriorityWeightPartition=10000
PriorityWeightJobSize=1000
PriorityWeightQOS=0
PriorityMaxAge=1-0
PriorityFavorSmall=YES
# COMPUTE NODES
NodeName=r1i[0-3]n[0-15] Sockets=2 CoresPerSocket=6 ThreadsPerCore=1 Procs=12
RealMemory=24054 Weight=1 Feature=ddr,wes,mem2
NodeName=r2i[0-3]n[0-15] Sockets=2 CoresPerSocket=6 ThreadsPerCore=1 Procs=12
RealMemory=24054 Weight=1 Feature=ddr,wes,mem2
NodeName=r3i[0-2]n[0-15] Sockets=2 CoresPerSocket=6 ThreadsPerCore=1 Procs=12
RealMemory=48294 Weight=2 Feature=qdr,wes,mem4,bigmem
NodeName=r3i3n[0-13] Sockets=2 CoresPerSocket=6 ThreadsPerCore=1 Procs=12
RealMemory=48294 Weight=2 Feature=qdr,wes,mem4,bigmem
NodeName=r4i[0-3]n[0-17] Sockets=2 CoresPerSocket=8 ThreadsPerCore=1 Procs=16
RealMemory=129073 Weight=3 Feature=fdr,san,mem8,bigmem
NodeName=r4i[4-7]n[0-17] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 Procs=24
RealMemory=128988 Weight=4 Feature=fdr,has,mem5,bigmem
NodeName=r5i[0-3]n[0-17] Sockets=2 CoresPerSocket=12 ThreadsPerCore=1 Procs=24
RealMemory=258268 Weight=5 Feature=fdr,bro,mem10,bigmem
#Partition Configurations
PartitionName=normal
Nodes=r[1-2]i[0-3]n[0-15],r3i[0-2]n[0-15],r3i3n[0-7],r4i[0-7]n[0-17],r5i[0-3]n[0-17]
Priority=10000 DefaultTime=04:00:00 MaxTime=08:00:00 State=UP Default=YES
PartitionName=idle
Nodes=r[1-2]i[0-3]n[0-15],r3i[0-2]n[0-15],r3i3n[0-7],r4i[0-7]n[0-17],r5i[0-3]n[0-17]
Priority=10 DefaultTime=04:00:00 MaxTime=08:00:00 State=UP
PartitionName=long
Nodes=r[1-2]i[0-3]n[0-15],r3i[0-2]n[0-15],r3i3n[0-7],r4i[0-7]n[0-17],r5i[0-3]n[0-17]
Priority=10000 DefaultTime=04:00:00 MaxTime=24:00:00 State=UP
AllowAccounts=root
PartitionName=debug Nodes=r3i3n[8-13]
