Raymond Hettinger added the comment: Attaching a draft patch:
* Repair the broken link to norobots-rfc.txt. * HTTP response codes >= 500 treated as a failed read rather than as a not found. Not found means that we can assume the entire site is allowed. A 5xx server error tells us nothing. * A successful read() updates the mtime (which is defined to be "the time the robots.txt file was last fetched"). * The can_fetch() method returns False unless we've had a read() with a 2xx or 4xx response. This avoids false positives in the case where a user calls can_fetch() before calling read(). ---------- assignee: -> rhettinger keywords: +patch Added file: http://bugs.python.org/file35215/fix_false_pos.diff _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue21469> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
