This should be helpful: http://www.robotstxt.org/wc/active/all.txt
On Monday 09 June 2003 07:36 pm, Steven Kallstrom wrote: > Dear List, > > So $_SERVER['HTTP_REFERER'] gives us the User Agent... is there anyway > to determine if in general the User-Agent is a bot, or do I have to > check for a specific bot... I want to strip out all GET information > from the URL so that bots can crawl my site session free... > > Thanks, > > SJK -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
