I've been working on eventlet-twisted integration and it includes a drop-in replacements for standard Python modules (including urllib and httplib) that do not block the reactor.
There was no release yet and there're no documentation written. (You can find the documentation for the original eventlet here http://wiki.secondlife.com/wiki/Eventlet, however, not all of it is applicable. You can also read the documentation of greenlet here http://codespeak.net/py/dist/greenlet.html) However, usage is very simple. 1. add this line before using any eventlet functions from eventlet.twistedutil import join_reactor 2. import eventlet.green.urllib instead of urllib 3. Use eventlet.twistedutil.deferToGreenThread function to make "blocking" functions from eventlet.green.urllib return Deferred. Here's an example on how to use eventlet.green.httplib and twisted's LineReceiver together to build a toy http proxy: http://devel.ag-projects.com/~denis/cgi-bin/hgweb.cgi/file/097d879e0228/examples/twisted_http_proxy.py To get the repository: hg clone http://devel.ag-projects.com/~denis/cgi-bin/hgweb.cgi/eventlet_twisted The only dependency is py.magic.greenlet from py library. The code is only tested on python2.5 linux & macos so it probably won't work on python2.4/windows but that will be fixed sometime in the future. On Sun, Dec 14, 2008 at 12:08 AM, Erik Wickstrom <[email protected]> wrote: > Hi all, > > I have an application that is doing some web spidering. Right now I'm > using urllib to retrieve the URLs, but it is painfully slow. I was > wondering if it's feasible to swap out urllib with a twisted client > that uses deferds so I can process urls in a more "parallel" fashion? > > I've done a bunch of Googleing, but I haven't come across anything > that I can use as a drop in replacement. If you can point me in the > right direction I'd really appreciate it! > > Thanks for your help! > Erik > > _______________________________________________ > Twisted-Python mailing list > [email protected] > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > _______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
