Thanks Niphlod - I appreciate the input and am definitely open to suggestions. I'm only running with Rocket right now but of course will be moving to a production server at some point. In my testing using multiprocessing to pull all 20 images at the same time is about 5-6 times faster than one after another, and my script returns in under a second. Just curious what would be an easier road for an I/O bound task like this?
On Tuesday, November 4, 2014 3:05:07 PM UTC-8, Niphlod wrote: > > launching external processes in a web environment is discouraged at so > many levels that basically you'll be left alone in the dark for any > debugging: very weird things will happen using the multiprocess package in > a threaded environment. Additional pains will be faced when web2py is run > by apache, nginx, uwsgi, etc etc etc. Even recent releases of python 3.x > are trying (hard) to address all the complications within a similar > environment, with a lot of machinery going around. > On python 2.x let's just say you're choosing a hard road for - practically > - no reason, since your task (fetching images) is I/O bound and not > cpu-bound. > > On Monday, November 3, 2014 7:03:23 PM UTC+1, Josh L wrote: >> >> Unfortunately after lots of experimentation I wasn't able to get the >> multiprocessing module to work with web2py. What I did find however was >> that I could use subprocess.check_output to launch a Python script >> containing the multiprocessing module and a pool of workers, and I can get >> data to the script by passing it command line arguments. The check_output >> function returns script results "print"ed to stdout which I could then >> parse back in the web2py process and use to update my database. So it looks >> like multiprocessing can work as long as you use it in an external script >> running in its own instance of python. Hope this helps someone! >> > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
