It’s always fun when I open my mouth in public only to turn it into a learning experience.
TL;DR version: Several enhancements to the script and to my PERL library to improve the accuracy were made. The now more accurate results aren’t very different. Details below: As a result of comments received in this thread and privately about the statistics pages, I started investigating the mysterious 5XX result codes and made several improvements to the script. First, I found the black text on blue hard to read, so I got rid of it. I converted the black text to white when the background is blue. Purely cosmetic, but still worth doing. Next, I started digging into why was LWP returning a 5xx result code. I discovered that it wasn’t getting a 5xx on the wire and was only sending one request which was getting a 3xx result and then it wasn’t sending an additional request. This led me to (erroneously) assume that it wasn’t attempting to follow the redirect. Some additional digging led me to the fact that LWP sometimes lies to you in both the documentation and the software. It was following redirects and continued to do so no matter how hard I tried to tell it not to. I did find out how to reverse the redirect back a step ($ua->previous() will return the response prior to the current response object in $ua if anyone cares). Armed with that information, I started looking at what I was getting and the text being reported by LWP with the 5xx errors. Turns out that I had neglected to install a module known as LWP::Protocol::https which meant that any redirect to HTTPS would fail with a 5xx result code from LWP without any packets over the wire being attempted. I’ve also made the script slightly more optimistic in that I do now count 3xx results as valid. This is now OVERLY optimistic in that anything that gets stuck at 3xx is actually a failed page load (the redirect went somewhere that didn’t actually work), but there are very few of these and they appear to relate to certificate verification failures which may be due to the version of root cert library on my system used by LWP more than anything else. The legitimate results post redirect are now guaranteed to come from an IPv6 destination because LWP is running with a source host name/address which does not have an A record or an IPv4 address associated. So… The revised statistics are now up and the results aren’t very startling. DNS results are unchanged. domain.name results are 82 (16.4%) up from 69 (13.8%). www.domain.name <http://www.domain.name/> results are 101 (20.2%) up from 81 (16.2%) So it only changed the results for 13-20 sites overall. speedtest.net <http://speedtest.net/> and wikimedia.org <http://wikimedia.org/> (but not www.wikimedia.org <http://www.wikimedia.org/>) failes (500) with “write failed: bad file descriptor” ??? Write? Interestingly, speedtest.net <http://speedtest.net/> has an AAAA record, but www.speedtest.net <http://www.speedtest.net/> does not. mega.nz <http://mega.nz/> still errors out 500 for timeout marca.com <http://marca.com/> still has a legitimate 500 error (timeout) A further enhancement to the script will probably replace the short status code in the table with the full status line for any result outside of the 2xx range. Owen
