Re: LinkWalker

[EMAIL PROTECTED] (Russell Coker) wrote in message news:<[EMAIL PROTECTED]>... > I have a nasty web spider with an agent name of "LinkWalker" downloading > everything on my site (including .tgz files). Does anyone know anything > about it? It's apparantly a link-validation robot operated by a

Re: LinkWalker

[EMAIL PROTECTED] (Russell Coker) wrote in message news:<[EMAIL PROTECTED]>... > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a file would prevent robots from visiting. Nope; see: http://www.robotstxt.org/wc/robots.html -- To UNSUBSCRIBE,

Re: LinkWalker

[EMAIL PROTECTED] (Russell Coker) wrote in message news:<[EMAIL PROTECTED]>... > I have a nasty web spider with an agent name of "LinkWalker" downloading > everything on my site (including .tgz files). Does anyone know anything > about it? It's apparantly a link-validation robot operated by a c

Re: LinkWalker

[EMAIL PROTECTED] (Russell Coker) wrote in message news:<[EMAIL PROTECTED]>... > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a file would prevent robots from visiting. Nope; see: http://www.robotstxt.org/wc/robots.html