: Cam Bazz
To: user@hive.apache.org
Sent: Tue, February 8, 2011 7:57:53 PM
Subject: filtering out crawlers
Hello,
Is there a practical way to filter the logs left by crawlers like google?
They usually have user-agent strings like
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com
Hello,
Is there a practical way to filter the logs left by crawlers like google?
They usually have user-agent strings like
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
is there a database for the
