Hi David, (Thanks for changing the subject to something more appropriate).
David Laehnemann <david.laehnem...@hhu.de> writes: > Yes, but only to an extent. The linked conversation ends with this: > >>> Do you have any best practice about setting MaxJobCount to a proper > number? > >> That depends upon your workload. You could probably set MaxJobCount > to at least 50000 with most systems (assuming you have at least a few > gigabytes of memory). Some sites run with a value of 1000000 or more. > > So, it is configurable. But this has a limit. And if you have lots of > users on a system submitting lots of jobs, even a value of 1000000 can > get exhausted. Yes, but start a lot more jobs and stay within the limit if you use jobs arrays. When you submit individual jobs, a job ID for each one needs to be written to the Slurm job database. This can cause the database to become unresponsive if the number submitted at one time, whether by snakemake or just a bash script looping over 'sbatch', is too high. If, on the other hand, you submit a job array, only one entry needs to be made in the database immediately, with entries for the elements of the array only being made when a job can actually start. This is why a large number of individual jobs with the same resource requirements prevents backfill from working properly. The mechanism only considers a certain (configurable) number of pending jobs to see whether they qualify for backfilling. In this context, a job array is counted as a single job, regardless of how large the array actually is. This will degrade the throughput of the system and thus negatively impact all users. Therefore, on our system we would not allow users to employ an mechanism which generates a large number jobs but does not employ job arrays. > And in either case, this is not something that speaks against a > workflow management system giving you additional control over things. > So I'm not sure what exactly we are arguing about, right here... I just wanted to point out that, whereas for some user approaches such as snakemake obviously scratch a very important itch, for people running HPC systems, and indeed for users who don't use such mechanisms, they may cause issues. Cheers, Loris > cheers, > david > > > > On Thu, 2023-02-23 at 17:41 +0100, Ole Holm Nielsen wrote: >> On 2/23/23 17:07, David Laehnemann wrote: >> > In addition, there are very clear limits to how many jobs slurm can >> > handle in its queue, see for example this discussion: >> > https://bugs.schedmd.com/show_bug.cgi?id=2366 >> >> My 2 cents: Slurm's job limits are configurable, see this Wiki page: >> https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#maxjobcount-limit >> >> /Ole >> -- Dr. Loris Bennett (Herr/Mr) ZEDAT, Freie Universität Berlin
