Using gnu-parallel with processes that have an extended startup time

2013-07-16 Thread Diaa Sami
Hi, I'm using gnu parallel with a custom python script that processes lines, one line in, one or more lines out, and this script happens to have a long startup time because of the kind of processing it has to perform on the input(it has to load a dictionary in memory first). I was wondering if g

Re: Using gnu-parallel with processes that have an extended startup time

2013-07-16 Thread Ole Tange
On Tue, Jul 16, 2013 at 2:20 PM, Diaa Sami wrote: > Hi, > I'm using gnu parallel with a custom python script that processes lines, one > line in, one or more lines out, and this script happens to have a long > startup time because of the kind of processing it has to perform on the > input(it h

Using gnu-parallel with processes that have an extended startup time

2013-07-16 Thread Diaa Sami
Hi, I'm using gnu parallel with a custom python script that processes lines, one line in, one or more lines out, and this script happens to have a long startup time because of the kind of processing it has to perform on the input(it has to load a dictionary in memory first). I was wondering if g

Re: Cannot use more than 16 remote servers

2013-07-16 Thread Nick Boutelier
Ole, using --ungroup worked! Thank you. The error after upping the ulimit still said "no more file handles" On Mon, Jul 15, 2013 at 7:04 AM, Ole Tange wrote: > What did the debugging output say as reason for not starting more jobs? > > /Ole > > On Sun, Jul 14, 2013 at 9:17 PM, Nick Boutelier