On our HPC systems we have a lot of users attempting to organize job arrays for varying purposes -- parameter scans, SSMD (Single-Script, Multiple Datasets). We eventually wrote an abstract utility to try to help them with the process:
https://github.com/jtfrey/job-templating-tool May be of some use to you. > On Jul 15, 2020, at 16:13 , c b <breedthoughts....@gmail.com> wrote: > > I'm trying to run an embarrassingly parallel experiment, with 500+ tasks that > all differ in one parameter. e.g.: > > job 1 - script.py foo > job 2 - script.py bar > job 3 - script.py baz > and so on. > > This seems like a case where having a slurm array hold all of these jobs > would help, so I could just submit one job to my cluster instead of 500 > individual jobs. It seems like sarray is only set up for varying an integer > index parameter. How would i do this for non-numeric values (say, if the > parameter I'm varying is a string in a given list) ? > >
