How far does GNU Parallel scale?

2014-08-20 Thread Ole Tange
I rarely run more than 200 jobs in parallel. But I was curious how many jobs I could safely run. This starts 3 sleeps: seq 10 | parallel -N1000 -j120 --pipe parallel -j0 sleep I have let those stay in the background for a day now, and the only issue I have had so far is that 'ps aux

How slow is GNU Parallel

2014-08-20 Thread Ole Tange
Back in the early days of GNU Parallel a job could be processed within just 3 ms. These days there are many more checks and much more flexibility has been added - and that costs. So I was curious how the performance of GNU Parallel has changed over time. www.gnu.org/s/parallel/process-time-j8-1.

Re: How slow is GNU Parallel

2014-08-20 Thread Achim Gratz
Ole Tange gnu.org> writes: > This was measured on my 4 hyperthreaded cpus (= 8 "cores") forced to > run 1.7GHz, so the timings may be lower if you have a faster cpu or if > you do not use hyperthreading. If you post the script, I could give you times for an Opteron 6328 on Cygwin on Windows Serve

Re: How slow is GNU Parallel

2014-08-20 Thread Ole Tange
On Wed, Aug 20, 2014 at 1:56 PM, Achim Gratz wrote: > Ole Tange gnu.org> writes: >> This was measured on my 4 hyperthreaded cpus (= 8 "cores") forced to >> run 1.7GHz, so the timings may be lower if you have a faster cpu or if >> you do not use hyperthreading. > > If you post the script, I could

Need a little help with multiple inputs

2014-08-20 Thread Eric Geoffroy
Hoi, GNU Parallel is so great I can hardly contain my excitement of finding it. Thanks to excellent documentation, examples, and videos, my peanut brain has been able to construct an almost working command. I have a dir of video files. The goal is to copy the videos from dir A, rename, and cop