New submission from larsfuse :
The standard (http://www.robotstxt.org/robotstxt.html) says:
> To allow all robots complete access:
> User-agent: *
> Disallow:
> (or just create an empty "/robots.txt" file, or don't use one at all)
Here I give python an empty file:
larsfuse added the comment:
> (...) refers users, for file structure, to
> http://www.robotstxt.org/orig.html. This says nothing about the effect of an
> empty file, so I don't see this as a bug.
That is incorrect. From that url you can find:
> The presence of an empty &
