On Thu, Apr 20, 2017 at 9:10 PM, Dima Pasechnik <dimp...@gmail.com> wrote: > > > On Thursday, April 20, 2017 at 4:04:19 PM UTC+1, Erik Bray wrote: >> >> On Thu, Apr 20, 2017 at 4:54 PM, Dima Pasechnik <dim...@gmail.com> wrote: >> > >> > >> > On Thursday, April 20, 2017 at 3:38:12 PM UTC+1, Erik Bray wrote: >> >> >> >> Just posting to say I've noticed and am investigating. >> >> >> >> It seems that somebody is performing a scrape of the front page which >> >> includes a list of users on the site [1] each of which is a link to a >> >> search query for all tickets and wiki pages associated with that user. >> >> That's what's causing the slowdown. >> >> >> >> The IP address seems to belong to Opera (maybe it's a VPN?) >> >> >> >> http://tools.tracemyip.org/lookup/107.167.122.77 >> > >> > note that the map on the right shows that the connection originates in >> > China. >> > Thus yes, it's a VPN. Actually, is sagemath.org blocked in China? I >> > don't >> > know. >> >> Strange, well, I created a firewall rule blocking that IP for now and >> everything seems to be fine. > > > Does this mean that we need some robots.txt somewhere, perhaps after some > restructuring, > which would protect expensive resources from this sort of overload?
There already is a robots.txt and this host was not respecting it. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
