You should be able to tell if it cares about robots.txt by looking in the logs to see if it's downloading /robots.txt. If it is then something like: User-agent: LinkWalker Disallow: /
will keep it off your site. If it doesn't, then iptables will keep it away. Robots info: http://www.global-positioning.com/robots_text_file/index.html The fact that it downloads binaries too makes me think it's a site sucker and not a legit spider. At 12:30 PM 12/23/01 -0800, Nick Jennings wrote: >On Sun, Dec 23, 2001 at 09:17:54PM +0100, Russell Coker wrote: >> >> I wasn't aware that there was any format to robots.txt, I thought that the >> mere presense of such a file would prevent robots from visiting. ---=<REMEMBER THE WORLD TRADE CENTER>=--- ___/`< WTC 911 >`\___ 00000100 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
