New submission from Raymond Hettinger: Resources:
* http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions * https://support.google.com/webmasters/answer/183669?hl=en * https://github.com/seomoz/reppy ---------- components: Library (Lib) keywords: easy messages: 218308 nosy: rhettinger priority: normal severity: normal status: open title: Support the Sitemap and Crawl-delay extensions in robotparser type: enhancement versions: Python 3.5 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue21475> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
