Sorry, but I think this conversation shouldn't continue.
This list is for system administrators, not for users with basic questions
about bash. People will unsubscribe if it goes on much longer.
On Thu, Jun 13, 2019 at 2:49 PM VG <gupta567va...@gmail.com> wrote:
> Hi Feng,
> I did something like this
>
> for i in *
> do
> if [ -d "$i" ]
> then cd "$i"
> a=$(ls *.tar.gz)
> echo $PWD/"$a"
> cd ..
> fi
> done
>
> This gave me the full path of my tar.gz files. Should I save this in a
> separate text file and then run an array script on it?
>
> Thanks
>
> Regards
> VARUN
>
> On Thu, Jun 13, 2019 at 1:46 PM Feng Zhang <prod.f...@gmail.com> wrote:
>
>> You can try to write the script to first scan all the files to get their
>> full path names and then run the Array jobs.
>>
>>
>> On Jun 13, 2019, at 1:20 PM, VG <gupta567va...@gmail.com> wrote:
>>
>> HI Joshua,
>> I like the array job option because essentially it will still be 1 job
>> and it will run them in parallel.
>>
>> I have one issue though. I can create an array script, but here I
>> presented a simple problem. Actually my individual tar.gz files are under
>> respective directories
>> For example
>> dir1 has file1.tar.gz
>> dir2 has file2.tar.gz
>> dir3 has file3.tar.gz
>>
>> The way I was then submitting them was
>>
>> for i in *
>> do
>> if [ -d "$i" ]
>> then cd "$i"
>> qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar -xf
>> *.tar"
>> cd ..
>> fi
>> done
>>
>> One way is I can pull out all the tar.gz in one folder and run array
>> script as you told, other wise is there a work around where everything runs
>> and also remains in the respective directories.
>>
>> Thanks for your help.
>>
>> Regards
>> Varun
>>
>>
>>
>>
>>
>> On Thu, Jun 13, 2019 at 1:11 PM Joshua Baker-LePain <j...@salilab.org>
>> wrote:
>>
>>> On Thu, 13 Jun 2019 at 9:32am, VG wrote
>>>
>>> > I have a scripting question regarding submitting jobs to the cluster.
>>> > There is a limitation per user of 1000 jobs only.
>>> > Let's say I have 1200 tar.gz files
>>> >
>>> > I tried to submit all the jobs together but after 1000 jobs it gave me
>>> an
>>> > error message saying per user limit is 1000 and after that it did not
>>> > queued the remaining jobs.
>>> > I want to write a script where if the submitted jobs goes below
>>> > 1000(because they finished running), then next jobs are submitted in
>>> the
>>> > queue. How can I do that?
>>> > I have written something like this:
>>> >
>>> > for i in *tar.gz
>>> > do
>>> > qsub -l h_vmem=4G -cwd -j y -b y -N tar -R y -q all.q,gpu.q "tar
>>> -xzf $i"
>>> > done
>>>
>>> The right answer to this problem (not the scripting issue, but how to
>>> untar all 1200 files without running afoul of the 1000 job limit) is an
>>> array job. You can submit 1 job with 1200 tasks to untar all the files.
>>> The relevant bits of the job script would include (assuming bash):
>>>
>>> targzs=(0 file1.tar.gz file2.tar.gz ... file1200.tar.gz)
>>> tar xzf ${targzs[$SGE_TASK_ID]}
>>>
>>> To submit it:
>>>
>>> qsub -t 1-1200 job.sh
>>>
>>> --
>>> Joshua Baker-LePain
>>> QB3 Shared Cluster Sysadmin
>>> UCSF
>>>
>> _______________________________________________
>> users mailing list
>> users@gridengine.org
>> https://gridengine.org/mailman/listinfo/users
>>
>> _______________________________________________
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
