Package: bugs.debian.org Access to Debian bug reports in Internet Archive is blocked by robots.txt http://web.archive.org/web/*/https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=736214
Current contents of https://bugs.debian.org/robots.txt: ----[cut here]---- User-Agent: Googlebot, bingbot, yandexbot, baiduspider, ia_archiver Allow: /cgi-bin/bugreport.cgi?bug= Allow: /cgi-bin/pkgreport.cgi?pkg=*;dist=unstable$ Disallow: /*/ User-agent: * Disallow: / ----[cut here]---- Looks like a problem is caused by User-Agent line, which lists user agents through comma instead of specifying every agent on separate line: ----[cut here]---- User-Agent: Googlebot User-Agent: bingbot User-Agent: yandexbot User-Agent: baiduspider User-Agent: ia_archiver Allow: /cgi-bin/bugreport.cgi?bug= Allow: /cgi-bin/pkgreport.cgi?pkg=*;dist=unstable$ Disallow: /*/ User-agent: * Disallow: / ----[cut here]---- Specifying each agent on separate line makes sense, because user-agent string can contain some special character and be pretty long. See section `3.2 File Format Description` for details: http://www.robotstxt.org/norobots-rfc.txt -- anatoly t. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
