On 2 Aug, 15:48, ma3mju <matt.u...@googlemail.com> wrote: > On 31 July, 11:34, MRAB <pyt...@mrabarnett.plus.com> wrote: > > > > > ma3mju wrote: > > > Hi all, > > > > I'm having trouble with multiprocessing I'm using it to speed up some > > > simulations, I find for large queues when the process reaches the > > > poison pill it does not exit whereas for smaller queues it works > > > without any problems. Has anyone else had this trouble? Can anyone > > > tell me a way around it? The code is in two files below. > > > [snip] > > > > #get number of cores and set the number on concurrent processes > > > num_hard_workers = 2 > > > num_workers = multiprocessing.cpu_count()*1.5 > > > easy_workers = [] > > > hard_workers = [] > > > #add poison pill for each worker and create the worker > > > for i in range(0,num_workers-num_hard_workers): > > > easy_work_queue.put(None) > > > easy_workers.append(multiprocessing.Process > > > (target=GP.RandomWalkGeneralizationErrorParallel,args= > > > (easy_work_queue,result_queue,))) > > > for i in range(0,num_hard_workers): > > > hard_work_queue.put(None) > > > hard_workers.append(multiprocessing.Process > > > (target=GP.RandomWalkGeneralizationErrorParallel,args= > > > (hard_work_queue,result_queue,))) > > > You have 2 hard workers and ceil(CPU_count * 1.5) - 2 easy workers. > > What if the number of CPUs was 1? That would give 2 hard and 0 easy! > > > Also, I recommend that you put only 1 'poison pill' in each queue and > > have the workers put it back when they see it. > > I'll give that a go in a sec and see if it helps. The processes quit > out for smaller queues though so it should in theory be alright. I'm > not too fussed about the CPU's it's only there because I change > between a uni PC and home one with a different number of cores in each > but both greater than one.
Just tried changing the poison pill part to no avail sadly -- http://mail.python.org/mailman/listinfo/python-list
