Re: urllib2 rate limiting

2008-01-12 Thread Dimitrios Apostolou
On Fri, 11 Jan 2008, Nick Craig-Wood wrote: > Here is an implementation based on that idea. I've used urllib rather > than urllib2 as that is what I'm familiar with. Thanks! Really nice implementation. However I'm stuck with urllib2 because of its extra functionality so I'll try to implement som

Re: urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
On Thursday 10 January 2008 22:42:44 Rob Wolfe wrote: > Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > > On Thu, 10 Jan 2008, Rob Wolfe wrote: > >> Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > >>> P.S. And something simpler: How can I disall

Re: urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
On Thu, 10 Jan 2008, Rob Wolfe wrote: > Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > >> P.S. And something simpler: How can I disallow urllib2 to follow >> redirections to foreign hosts? > > You need to subclass `urllib2.HTTPRedirectHandler`, override > `ht

urllib2 rate limiting

2008-01-10 Thread Dimitrios Apostolou
Hello list, I want to limit the download speed when using urllib2. In particular, having several parallel downloads, I want to make sure that their total speed doesn't exceed a maximum value. I can't find a simple way to achieve this. After researching a can try some things but I'm stuck on th

Re: urllib2 disable proxy

2008-01-03 Thread Dimitrios Apostolou
On Wed, 2 Jan 2008, Rob Wolfe wrote: > Dimitrios Apostolou <[EMAIL PROTECTED]> writes: > >> Hello list, >> >> I've been looking for a way to explicitly disable the use of proxies with >> urllib2, no matter what the environment dictates. Unfortunately I

urllib2 disable proxy

2008-01-02 Thread Dimitrios Apostolou
Hello list, I've been looking for a way to explicitly disable the use of proxies with urllib2, no matter what the environment dictates. Unfortunately I can't find a way in the documentation, and reading the source leads me to believe that something like the following does the job: req.set_pr