On Jan 19, 10:26 am, Adam Tauno Williams <awill...@opengroupware.us> wrote: > > I decided to play around with the multiprocessing module, and I'm > > having some strange side effects that I can't explain. It makes me > > wonder if I'm just overlooking something obvious or not. Basically, I > > have a script parses through a lot of files doing search and replace > > on key strings inside the file. I decided the split the work up on > > multiple processes on each processor core (4 total). I've tried many > > various ways doing this form using pool to calling out separate > > processes, but the result has been the same: computer crashes from > > endless process spawn. > > Are you hitting a ulimit error? The number of processes you can create > is probably limited. > > TIP: close os.stdin on your subprocesses. > > > > > Here's the guts of my latest incarnation. > > def ProcessBatch(files): > > p = [] > > for file in files: > > p.append(Process(target=ProcessFile,args=file)) > > for x in p: > > x.start() > > for x in p: > > x.join() > > p = [] > > return > > Now, the function calling ProcessBatch looks like this: > > def ReplaceIt(files): > > processFiles = [] > > for replacefile in files: > > if(CheckSkipFile(replacefile)): > > processFiles.append(replacefile) > > if(len(processFiles) == 4): > > ProcessBatch(processFiles) > > processFiles = [] > > #check for left over files once main loop is done and process them > > if(len(processFiles) > 0): > > ProcessBatch(processFiles) > > According to this you will create files is sets of four, but an unknown > number of sets of four.
What would be the proper way to only do a set of 4, stop, then do another set of 4? I'm trying to only 4 files at time before doing another set of 4. -- http://mail.python.org/mailman/listinfo/python-list
