Suppose that I have a multi-core computer with N CPU's, and I create a multiprocessing.Pool in Python 2.6 with N-1 Processes.
(My rationale for choosing N-1 Processes was discussed here: http://groups.google.com/group/comp.lang.python/browse_frm/thread/65ba3ccd4be8228c) Then I use Pool.map_async() to break up a long-running task T1 into M chunks, where M > 2N. Suppose that I have a second, parallelizable, long-running task T2 that I want to address in REAL TIME when the need arises. Using Pool, is there a way for me to insert the chunks of T2 at the HEAD of the task queue, instead of at its TAIL? I've been snooping around inside Pool, and I would guess that what I want to do is to manipulate Pool._inqueue, which is a multiprocessing.queues.SimpleQueue object. I haven't found any documentation for SimpleQueue. It appears to have only the most rudimentary of public methods -- put, get, and empty. Reading further, I can see that multiprocessing.Queue (not the same as SimpleQueue) has put_nowait(), which looks like what I need. Assuming that I had access to put_nowait(), then I would need an option in map_async() which would invoke put_nowait() rather than just plain put(). Does anyone know of an efficient way to accomplish this? I'm thinking that a subclass of Pool which replaces the SimpleQueue with a full- fledged Queue, and which overrides map_async() allowing for a no-wait task, would be very useful... but maybe I'm re-inventing the wheel. Thanks for your advice! -- http://mail.python.org/mailman/listinfo/python-list
