On Thu, 2010-01-21 at 07:13 -0500, Neal Becker wrote: > I'm using multiprocessing as a poor man's batch system. It is working OK, > except that it doesn't seem to do load balancing quite right. > I have a 8-cpu machine. If I start, using multiprocessing pool, calling map > with more than 8 elements, it will start 8 processes. It seems, though, > that when some of them finish, it does not start others right away. The > load average may be a lot less than 8 but still additional processes are not > active.
If your processes are I/O bound your system will never appear to be saturated. Multiple pending processes and a low load average usually indicated waiting on some other resource. > Is there a way to use multiprocessing to achieve the effect I want? (I > really want a simple batch system, but without complex setup) Well, honestly, that isn't possible (unless you just choose to ignore all the things that can go wrong: crashed worked, hung workers, etc...) -- OpenGroupware developer: awill...@whitemice.org <http://whitemiceconsulting.blogspot.com/> OpenGroupare & Cyrus IMAPd documenation @ <http://docs.opengroupware.org/Members/whitemice/wmogag/file_view> -- http://mail.python.org/mailman/listinfo/python-list
