Hi Hugh,
> By the way, have I got my robots.txt right?
In particular, is the
User-agent: LDSpider
correct?
Should I worry about case-sensitivity?
The library (norbert) that is employed in LDspider is case-insensitive
for the user agent. The user agent that is sent is "ldspider".
I suppose you want ldspider to crawl your site (highly appreciated), so
you should change the line in your robots.txt for LDspider to:
a) Disallow:
b) Allow: /
And not leave it with:
c) Allow: *
The star there does not bring the desired behaviour (and I have not
found it in the spec for the path either), in fact, it keeps LDspider
from crawling the folders you specified for exclusion for the other
crawlers.
Cheers,
Tobias
