multiprocessing module in async db query
This looks like a tornado problem, but trust me, it is almost all about the mechanism of multiprocessing module. I borrowed the idea from http://gist.github.com/312676 to implement an async db query web service using tornado. p = multiprocessing.Pool(4) class QueryHandler(tornado.web.RequestHandler): ... @tornado.web.asynchronous def get(self): ... p.apply_async(async_func, [sql_command, arg1, arg2, arg3, ], callback_func) def callback_func(self, data): self.write(data) def async_func(sql_command, arg1, arg2, arg3): ''' do the actual query job ''' ... # data is the query result by executing sql_command return data So the workflow is like this, get() --> fork a subprocess to process the query request in async_func() -> when async_func() returns, callback_func uses the return result of async_func as the input argument, and send the query result to the client. So the problem is the the query result as the result of sql_command might be too big to store them all in the memory, which in our case is stored in the variable "data". Can I send return from the async method early, say immediately after the query returns with the first result set, then stream the results to the browser. In other words, can async_func somehow notify callback_func to prepare receiving the data before async_func actually returns? -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module in async db query
Hi Philip, multiprocessing.Queue is used to transfer data between processes, how it could be helpful for solving my problem? Thanks! Sheng On Mar 8, 6:34 pm, Philip Semanchuk wrote: > On Mar 8, 2011, at 3:25 PM, Sheng wrote: > > > This looks like a tornado problem, but trust me, it is almost all > > about the mechanism of multiprocessing module. > > [snip] > > > So the workflow is like this, > > > get() --> fork a subprocess to process the query request in > > async_func() -> when async_func() returns, callback_func uses the > > return result of async_func as the input argument, and send the query > > result to the client. > > > So the problem is the the query result as the result of sql_command > > might be too big to store them all in the memory, which in our case is > > stored in the variable "data". Can I send return from the async method > > early, say immediately after the query returns with the first result > > set, then stream the results to the browser. In other words, can > > async_func somehow notify callback_func to prepare receiving the data > > before async_func actually returns? > > Hi Sheng, > Have you looked at multiprocessing.Queue objects? > > HTH > Philip -- http://mail.python.org/mailman/listinfo/python-list
IDLE python shell freezes after running show() of matplotlib
I am having a weird problem on IDLE. After I plot something using show () of matplotlib, the python shell prompt in IDLE just freezes that I cannot enter anything and there is no new ">>>" prompt show up. I tried ctrl - C and it didn't work. I have to restart IDLE to use it again. My system is Ubuntu Linux 9.04. I used apt-get to install IDLE. -- http://mail.python.org/mailman/listinfo/python-list
