On 26/12/2018 19:04, Terry Reedy wrote:
On 12/26/2018 10:35 AM, Simon Connah wrote:
Hi,
I want to build a simple web crawler. I know how I am going to do it
but I have one problem.
Obviously I don't want to negatively impact any of the websites that I
am crawling so I want to implement some
On 26/12/2018 18:30, Richard Damon wrote:
On 12/26/18 10:35 AM, Simon Connah wrote:
Hi,
I want to build a simple web crawler. I know how I am going to do it
but I have one problem.
Obviously I don't want to negatively impact any of the websites that I
am crawling so I want to implement some fo
On 12/26/2018 10:35 AM, Simon Connah wrote:
Hi,
I want to build a simple web crawler. I know how I am going to do it but
I have one problem.
Obviously I don't want to negatively impact any of the websites that I
am crawling so I want to implement some form of rate limiting of HTTP
requests
On 12/26/18 10:35 AM, Simon Connah wrote:
> Hi,
>
> I want to build a simple web crawler. I know how I am going to do it
> but I have one problem.
>
> Obviously I don't want to negatively impact any of the websites that I
> am crawling so I want to implement some form of rate limiting of HTTP
> req
Hi,
I want to build a simple web crawler. I know how I am going to do it but
I have one problem.
Obviously I don't want to negatively impact any of the websites that I
am crawling so I want to implement some form of rate limiting of HTTP
requests to specific domain names.
What I'd like is
