On Mar 9, 2011, at 10:22 AM, Sheng wrote: > Hi Philip, > > multiprocessing.Queue is used to transfer data between processes, how > it could be helpful for solving my problem? Thanks!
I misunderstood -- I thought transferring data between processes *was* your problem. If both of your functions are in the same process, I don't understand how multiprocessing figures into it at all. If you want a function to start returning results before that function completes, and you want those results to be processed by other code *in the same process*, then you'll have to use threads. A Queue object for threads exists in the standard library too. You might find that useful. HTH Philip > > On Mar 8, 6:34 pm, Philip Semanchuk <phi...@semanchuk.com> wrote: >> On Mar 8, 2011, at 3:25 PM, Sheng wrote: >> >>> This looks like a tornado problem, but trust me, it is almost all >>> about the mechanism of multiprocessing module. >> >> [snip] >> >>> So the workflow is like this, >> >>> get() --> fork a subprocess to process the query request in >>> async_func() -> when async_func() returns, callback_func uses the >>> return result of async_func as the input argument, and send the query >>> result to the client. >> >>> So the problem is the the query result as the result of sql_command >>> might be too big to store them all in the memory, which in our case is >>> stored in the variable "data". Can I send return from the async method >>> early, say immediately after the query returns with the first result >>> set, then stream the results to the browser. In other words, can >>> async_func somehow notify callback_func to prepare receiving the data >>> before async_func actually returns? >> >> Hi Sheng, >> Have you looked at multiprocessing.Queue objects? >> >> HTH >> Philip > > -- > http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
