RE: howto stop crawler and bots according to their user agent string

2008-07-15 Thread Mathias Walter
Hi, > > I've put a robots.txt in webapps/ROOT, but this file is not > > read again. > > So, to check, the crawlers are not reading your robots.txt > and are crawling your site anyway? I don't know it exactly. The problem is that the sites are linked from anywhere. I'm not sure, if a crawler tha

howto stop crawler and bots according to their user agent string

2008-07-15 Thread Mathias Walter
their useragent string. -- Mit freundlichen Grüßen, Mathias Walter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]