On Sat, 13 Dec 2008 10:08:43 -0800, Erik Wickstrom <[email protected]>
wrote:
Hi all,
I have an application that is doing some web spidering. Right now I'm
using urllib to retrieve the URLs, but it is painfully slow. I was
wondering if it's feasible to swap out urllib with a twisted client
that uses deferds so I can process urls in a more "parallel" fashion?
I've done a bunch of Googleing, but I haven't come across anything
that I can use as a drop in replacement. If you can point me in the
right direction I'd really appreciate it!
There is no drop-in replacement. There is an HTTP client in Twisted,
`twisted.web.client´. There is also a more featureful HTTP client in
development but not yet available in a release of Twisted. You can
find an example of how to use the former in the Twisted Web examples,
<http://twistedmatrix.com/projects/web/documentation/examples/>, and
you can follow progress of the latter at
<http://twistedmatrix.com/trac/ticket/886>.
Jean-Paul
_______________________________________________
Twisted-Python mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
