Re: [PHP] robot.txt file

Everything you need to know about robots.txt files but were afraid to ask http://www.robotstxt.org/wc/robots.html __ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail http://personal.mail.yahoo.com/ -- PHP General Mailing List (ht

Re: [PHP] robot.txt file

Hi Don, How do you work this out? ie how do you know when you are spidered? I'm interested in what logs and what patterns lead you to know this? Thanks, Tom > > On 07-Jul-01 Mark Lo wrote: > > Hi, > > > >I don't know how to write a robot.txt file. Can anyone send me a > > link or a sa