to recognize "executables" independent of the OS and file
extensions and associations? If that's true, perhaps that might lead to some
solution to your problem.
Mark
---- Original Message ----
Subject: [EMAIL PROTECTED] Blocking crawling of CGIs
From: Tony Rice (trice) <[E
We've had some instances where crawlers have stumbled onto a cgi script
which refers to itself and start pounding the server with requests to
that cgi.
There are so many CGI scripts on this server that I don't want to
maintain a huge robots.txt file. Any suggestions on other techniques to
keep cr
