I don’t understand why you need a job to launch other jobs? Your scheduler is smart enough to submit all those jobs.
Why not #!/bin/bash #SBATCH --nodes=1 #SBATCH --ntasks=12 # number of samples #SBATCH --mem-per-cpu=1G #SBATCH --time=0-00:30:00 # 30 minutes #SBATCH --output=my.stdout #SBATCH --job-name="just_a_test" srun python run_sgRNA.py $1 `basedir $1` #Sbatch has allocated the tasks, python can use 12 tasks safely. Then on your scheduling server just do like you did $for sdir in /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_*; do sbatch my_batch_script.sbatch $sdir; done Jordan > On Aug 2, 2016, at 8:34 PM, Lachlan Musicman <[email protected]> wrote: > > Sometimes we would like to run jobs in parallel without using arrays because > the files aren't well named. But the files are all in the same folder. > > We have written a small script that loops over each file, constructs the > command in question and runs it. > > We only want each command to run once, but we would like to launch them all > at the same time - from the same sbatch file if possible. We are using srun > -n 1 to get this result. > > The script looks like this > > #!/bin/bash > > #SBATCH --nodes=1 > #SBATCH --ntasks=12 # number of samples > #SBATCH --mem-per-cpu=1G > #SBATCH --time=0-00:30:00 # 30 minutes > #SBATCH --output=my.stdout > #SBATCH --job-name="just_a_test" > > for sdir in /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_*; do > samp=`basename $sdir` > fq=$sdir/*gz > cmd="python run_sgRNA.py $fq $samp" > srun -n 1 $cmd & # -n 1 to run the command once only (...how to > specify more than 1 threads per task???) > done > > wait > > > > So my questions are: > > - is the above the best way to deal with this scenario? > > - is getting each srun to use multiple threads (but only run once) just a > matter of using -c in srun? eg to execute each of twelve jobs once, but with > four threads: srun -n 1 -c 4 $cmd & > > > Cheers > L. > > > ------ > The most dangerous phrase in the language is, "We've always done it this way." > > - Grace Hopper
