max_array_tasks Specify the maximum number of tasks that be included in a job array. The default limit is MaxArraySize, but this option can be used to set a lower limit. For example, max_array_tasks=1000 and MaxArraySize=100001 would permit a maximum task ID of 100000, but limit the number of tasks in any single job array to 1000. https://slurm.schedmd.com/slurm.conf.html <https://slurm.schedmd.com/slurm.conf.html>
SchedulerParameters=max_array_tasks=1000 MaxArraySize=100000 See commit: https://github.com/SchedMD/slurm/commit/09c13fb292a4a6a56b4078de840aae0d4db70309 <https://github.com/SchedMD/slurm/commit/09c13fb292a4a6a56b4078de840aae0d4db70309> -- Merlin Hartley Computer Officer MRC Mitochondrial Biology Unit University of Cambridge Cambridge, CB2 0XY United Kingdom > On 26 Feb 2019, at 14:27, Jeffrey Frey <f...@udel.edu> wrote: > > Also see "https://slurm.schedmd.com/slurm.conf.html > <https://slurm.schedmd.com/slurm.conf.html>" for MaxArraySize/MaxJobCount. > > We just went through a user-requested adjustment to MaxArraySize to bump it > from 1000 to 10000; as the documentation states, since each index of an array > job is essentially "a job," you must be sure to also adjust MaxJobCount (from > 10000 to 100000 in our case). Adjusting MaxJobCount requires a restart of > slurmctld; though the documentation doesn't state it, so does adjustment of > MaxArraySize (scontrol reconfigure will succeed but leave the previous limit > in effect, see "https://bugs.schedmd.com/show_bug.cgi?id=6553 > <https://bugs.schedmd.com/show_bug.cgi?id=6553>"). > > The "MaxArraySize" is a bit of a misnomer since it's really 1 + the top of > the valid range of indices -- "MaxArrayIndex" would be more apt. Our users > were very happy with Grid Engine's allowance of any index range and striding > that produces no more than "max_aj_tasks" indices; since moving to Slurm > they're forced to come up with their own index-mapping functionality at > times, but the relatively low MaxArraySize versus what we had in GridEngine > (75000) has been especially frustrating for them. > > So far the 10000/100000 combo hasn't come close to exhausting resources on > our slurmctld nodes; but we haven't actually submitted a couple 10000-index > array jobs and enough other jobs to hit 100000 active jobs, so current memory > usage isn't an adequate measure of usage under load. Since the slurm.conf > documentation states: > > > Performance can suffer with more than a few hundred thousand jobs. > > > we're reluctant to increase MaxJobCount too much higher. > > > > >> On Feb 26, 2019, at 3:18 AM, Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk >> <mailto:ole.h.niel...@fysik.dtu.dk>> wrote: >> >> On 2/26/19 9:07 AM, Marcus Wagner wrote: >>> Does anyone know, why per default the number of array elements is limited >>> to 1000? >>> We have one user, who would like to have 100k array elements! >>> What is more difficult for the scheduler, one array job with 100k elements >>> or 100k non-array jobs? >>> Where did you set the limit? Do your users use array jobs at all? >> >> Google is your friend :-) >> >> https://slurm.schedmd.com/job_array.html >> <https://slurm.schedmd.com/job_array.html> >> >>> A new configuration parameter has been added to control the maximum job >>> array size: MaxArraySize. The smallest index that can be specified by a >>> user is zero and the maximum index is MaxArraySize minus one. The default >>> value of MaxArraySize is 1001. The maximum MaxArraySize supported in Slurm >>> is 4000001. Be mindful about the value of MaxArraySize as job arrays offer >>> an easy way for users to submit large numbers of jobs very quickly. >> >> /Ole >> > > > :::::::::::::::::::::::::::::::::::::::::::::::::::::: > Jeffrey T. Frey, Ph.D. > Systems Programmer V / HPC Management > Network & Systems Services / College of Engineering > University of Delaware, Newark DE 19716 > Office: (302) 831-6034 Mobile: (302) 419-4976 > :::::::::::::::::::::::::::::::::::::::::::::::::::::: > > > >
