Here is the link for GNU parallel https://www.gnu.org/software/parallel/parallel_tutorial.html <https://www.gnu.org/software/parallel/parallel_tutorial.html>
Shenglong > On Aug 19, 2016, at 7:27 AM, Shenglong Wang <[email protected]> wrote: > > For this kind of short time jobs, maybe they are try to use GNU parallel. Ask > a few compute nodes from job scheduler, use parallel to run many jobs at the > same time. parallel will launch new jobs when there are CPU cores free. > > Best, > Shenglong > >> On Aug 19, 2016, at 5:47 AM, Gokcehan Kara <[email protected]> wrote: >> >> Hello, >> >> We have a small cluster consisting of ~40 machines in our university for >> running MD simulations. One of our users is working on a project where she >> needs to submit many jobs of about 5 min lengths. To my understanding, she >> is doing some calculations in a script to find some parameters and submit 4 >> or 5 of these jobs at the same time in the script. And then rinse and >> repeat. So the problem is that she says after a while (~3000 jobs) jobs >> start to take some time before scheduled. We thought it might be related to >> some slurm settings about fairness or such but couldn't find anything about >> it. Is there a way to disable this waiting time? >> >> Thanks in advance, >> Gokcehan >
