R. David Murray added the comment:

Lucaz pointed out on IRC that the problem is that the current robotparser is 
implementing an outdated robots.txt standard.  He may work on fixing that.

----------

_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue17403>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to