On Thu, 13 Jun 2019 at 9:32am, VG wrote
I have a scripting question regarding submitting jobs to the cluster.
There is a limitation per user of 1000 jobs only.
Let's say I have 1200 tar.gz files
I tried to submit all the jobs together but after 1000 jobs it gave me an
error message saying per user limit is 1000 and after that it did not
queued the remaining jobs.
I want to write a script where if the submitted jobs goes below
1000(because they finished running), then next jobs are submitted in the
queue. How can I do that?
I have written something like this:
for i in *tar.gz
do
qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xzf $i"
done
The right answer to this problem (not the scripting issue, but how to
untar all 1200 files without running afoul of the 1000 job limit) is an
array job. You can submit 1 job with 1200 tasks to untar all the files.
The relevant bits of the job script would include (assuming bash):
targzs=(0 file1.tar.gz file2.tar.gz ... file1200.tar.gz)
tar xzf ${targzs[$SGE_TASK_ID]}
To submit it:
qsub -t 1-1200 job.sh
--
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
