On Sat, May 14, 2011 at 6:12 AM, Gopal Patel <[email protected]> wrote: > there is exabot crawler crawling my unfinished website continuously....what > can I do to stop it ? I have disallowed everything in robot.txt .
That should be robots.txt, with an 's'. They claim it respects robots.txt: http://www.exalead.com/software/services/knowledgebase/websearchengine/webmasterguide/#how-may-i-exclude-my-site-from-being-crawled-by-the-exalead-robot But it seems to be a badly behaved bot in general, so you may want to add it's IP addresses to your dos.yaml file: http://code.google.com/appengine/docs/python/config/dos.html (This only blocks access to dynamic pages) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
