Terry J. Reedy <tjre...@udel.edu> added the comment:
https://docs.python.org/2.7/library/robotparser.html#module-robotparser and https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser refers users, for file structure, to http://www.robotstxt.org/orig.html. This says nothing about the effect of an empty file, so I don't see this as a bug. Even if it was, I would be dubious about reversing the effect without a deprecation notice first, and definitely not in 2.7. I would propose instead that the doc be changed to refer to the new file, with more and better examples, but add a note that robotparser interprets empty files as 'block all' rather than 'allow all'. Try bringing this up on python-ideas. ---------- nosy: +terry.reedy stage: -> test needed type: behavior -> enhancement versions: +Python 3.8 -Python 2.7 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue35457> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
