This should be helpful:
http://www.robotstxt.org/wc/active/all.txt
On Monday 09 June 2003 07:36 pm, Steven Kallstrom wrote:
> Dear List,
>
> So $_SERVER['HTTP_REFERER'] gives us the User Agent... is there anyway
> to determine if in general the User-Agent is a bot, or do I have to
> check for a
Dear List,
So $_SERVER['HTTP_REFERER'] gives us the User Agent... is there anyway
to determine if in general the User-Agent is a bot, or do I have to
check for a specific bot... I want to strip out all GET information
from the URL so that bots can crawl my site session free...
Thanks,
SJK
-
2 matches
Mail list logo
