New submission from Skyreka Agence Web <ad...@skyreka.com>:
I can't documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.skyreka.com/robots.txt, I believe that this is how it's implemented now in most parsers ? ---------- messages: 377055 nosy: admin2 priority: normal severity: normal status: open title: Robots.txt _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue41803> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
