On Tue, 7 Aug 2001, Ian Perry wrote: > Oh damn... looking at the logs.... looks like here comes another one... > "GET /robots.txt HTTP/1.0"... repeat.
That's usually from a search engine. robots.txt is an (advisory) control method so that search engines don't try to index, for example, dynamically generated or password protected content. Unless there's a new worm out that uses that somehow (ugh).
