Of course! Thankyou. ------ The most dangerous phrase in the language is, "We've always done it this way."
- Grace Hopper On 3 August 2016 at 13:52, Jordan Willis <[email protected]> wrote: > I don’t understand why you need a job to launch other jobs? Your scheduler > is smart enough to submit all those jobs. > > Why not > > #!/bin/bash > > #SBATCH --nodes=1 > #SBATCH --ntasks=12 # number of samples > #SBATCH --mem-per-cpu=1G > #SBATCH --time=0-00:30:00 # 30 minutes > #SBATCH --output=my.stdout > #SBATCH --job-name="just_a_test" > > > > srun python run_sgRNA.py $1 `basedir $1` #Sbatch has allocated the tasks, > python can use 12 tasks safely. > > > > Then on your scheduling server just do like you did > > $for sdir in /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_*; do > sbatch my_batch_script.sbatch $sdir; done > > > Jordan > > On Aug 2, 2016, at 8:34 PM, Lachlan Musicman <[email protected]> wrote: > > Sometimes we would like to run jobs in parallel without using arrays > because the files aren't well named. But the files are all in the same > folder. > > We have written a small script that loops over each file, constructs the > command in question and runs it. > > We only want each command to run once, but we would like to launch them > all at the same time - from the same sbatch file if possible. We are using > srun -n 1 to get this result. > > The script looks like this > > #!/bin/bash > > #SBATCH --nodes=1 > #SBATCH --ntasks=12 # number of samples > #SBATCH --mem-per-cpu=1G > #SBATCH --time=0-00:30:00 # 30 minutes > #SBATCH --output=my.stdout > #SBATCH --job-name="just_a_test" > > for sdir in /pipeline/Runs/ProjectFolders/Project_Michael-He/Sample_*; do > samp=`basename $sdir` > fq=$sdir/*gz > cmd="python run_sgRNA.py $fq $samp" > srun -n 1 $cmd & # -n 1 to run the command once only (...how > to specify more than 1 threads per task???) > done > > wait > > > > So my questions are: > > - is the above the best way to deal with this scenario? > > - is getting each srun to use multiple threads (but only run once) just a > matter of using -c in srun? eg to execute each of twelve jobs once, but > with four threads: srun -n 1 -c 4 $cmd & > > > Cheers > L. > > > ------ > The most dangerous phrase in the language is, "We've always done it this > way." > > - Grace Hopper > > >
