On Thu, 11 May 2017 12:18 pm, liyucun2...@gmail.com wrote: > Hi Everyone, > > Thanks for stoping by. I am working on a feature to crawl website content > every 1 min. I am curious to know if there any good open source project > for this specific scenario.
I agree with Iuri: crawling a website every minute is abuse. Unless it is your own website, once a month is more appropriate -- and even then, you should be very careful to restrict the rate at which you make requests. -- Steve Emoji: a small, fuzzy, indistinct picture used to replace a clear and perfectly comprehensible word. -- https://mail.python.org/mailman/listinfo/python-list
