ooooh!
Thankyou! ------ The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 3 August 2016 at 13:42, Gene Soudlenkov <[email protected]> wrote: > > Actually, you can easily use arrays: > > arr=( /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_* ) > > samp=`basename ${arr[$SLURM_ARRAY_TASK_ID]}` > fq=$sdir/*gz > cmd="python run_sgRNA.py $fq $samp" > srun -n 1 $cmd & > > > Cheers, > Gene > > -- > New Zealand eScience Infrastructure > Centre for eResearch > The University of Auckland > e: [email protected] > p: +64 9 3737599 ext 89834 c: +64 21 840 825 f: +64 9 373 7453 > w: www.nesi.org.nz > > > On 3/08/16 3:33 pm, Lachlan Musicman wrote: > >> Sometimes we would like to run jobs in parallel without using arrays >> because the files aren't well named. But the files are all in the same >> folder. >> >> We have written a small script that loops over each file, constructs the >> command in question and runs it. >> >> We only want each command to run once, but we would like to launch them >> all at the same time - from the same sbatch file if possible. We are >> using srun -n 1 to get this result. >> >> The script looks like this >> >> #!/bin/bash >> >> #SBATCH --nodes=1 >> #SBATCH --ntasks=12 # number of samples >> #SBATCH --mem-per-cpu=1G >> #SBATCH --time=0-00:30:00 # 30 minutes >> #SBATCH --output=my.stdout >> #SBATCH --job-name="just_a_test" >> >> for sdir in /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_*; do >> samp=`basename $sdir` >> fq=$sdir/*gz >> cmd="python run_sgRNA.py $fq $samp" >> srun -n 1 $cmd & # -n 1 to run the command once only >> (...how to specify more than 1 threads per task???) >> done >> >> wait >> >> >> >> So my questions are: >> >> - is the above the best way to deal with this scenario? >> >> - is getting each srun to use multiple threads (but only run once) just >> a matter of using -c in srun? eg to execute each of twelve jobs once, >> but with four threads: srun -n 1 -c 4 $cmd & >> >> >> Cheers >> L. >> >> >> ------ >> The most dangerous phrase in the language is, "We've always done it this >> way." >> >> - Grace Hopper >> >
