On Sep 15, 3:14 pm, Chris Angelico <ros...@gmail.com> wrote: > On Fri, Sep 16, 2011 at 6:52 AM, John Ladasky <lada...@my-deja.com> wrote: > > Suppose that I have a second, parallelizable, long-running task T2 > > that I want to address in REAL TIME when the need arises. Using Pool, > > is there a way for me to insert the chunks of T2 at the HEAD of the > > task queue, instead of at its TAIL? > > That's a self-contradiction there. Even if you insert a task into the > head of the queue, it won't be addressed in real time; the only way to > do that would be to have a dedicated process, always ready to handle > the real-time events, or else some kind of interrupt system. > > But if you just want them handled when there's a worker available, > what you're after is a priority queue system. I don't know if one > exists in Python already or not, but if not, it'd be a worthwhile > addition. > > ChrisA
Hi, Chris, Sorry if I'm not quite familiar with the proper terminology for queues. Let me try to define what I need as precisely as I can. 1) I aim to keep all five child Processes as busy as possible at all times, crunching numbers. 2) I break my data for the low-priority task, T1, into large enough chunks to benefit from multiprocessing, but small enough so that any given Process should become available fairly frequently -- say, every 50 milliseconds. Starting 50 milliseconds late would be close enough to "real time" for my purposes. 3) But let's say that T1 still has twenty chunks in the Process queue when T2 comes along. I could be waiting 300 milliseconds for the queue to empty. If I could just ensure that T2 gets the next available Processes, I would be happy. Since T2 is also parallelizable and I am using five child Processes, I would divide T2 into exactly five chunks. These chunks should get inserted at positions 0-4 in the queue rather than at 20-24. T2 would briefly commandeer all five Processes and then clear out. If that's not putting T2 at the head of the queue, I guess I don't know a better way to describe it. -- http://mail.python.org/mailman/listinfo/python-list
