On Apr 4, 7:21 pm, Carl Witty <carl.wi...@gmail.com> wrote:
> On Sat, Apr 4, 2009 at 6:53 PM, mabshoff <mabsh...@googlemail.com> wrote:
>
> > Hello folks,
>
> > for the last couple hours Sage's trac has been very slow or completely
> > unresponsive. This is caused by the MS as well as Yahoo search engines
> > hammering the website (as can be seen by the proxy error log as well
> > as trac's log). It is non-obvious to me how to fix this, i.e. the
> > tracd does not support a robots.txt out of the box for example.
>
> Well, trac.sagemath.org claims to be served by apache; I'm guessing
> this means that apache is serving as a proxy?
Yes, apache proxies to the tracd which is running on sagemath.org:
9123.
> (You mentioned the
> "proxy error log".) Can apache be configured to serve its own
> robots.txt and proxy everything else?
Hmm, I am not sure. There is an extension to trac that let's one
create Robots.txt in the trac wiki and that is then served on
request.
Right now http://trac.sagemath.org/robots.txt returns "Environment not
found".
I have thought of various ways how to deal with this, i.e. the main
problem seems to be that the timeline can now be accessed at any point
you want and each ticket has a link to jump to the the point of the
timeline when it was created. This is quite expensive for the database
to construct and the problem with slowdown seems to be due to sqlite
not being able to keep up and much less due to lack of CPU or RAM
since sagemath.org just sits there not doing much. So we could either
disallow the robots to follow timeline links which is likely good
enough to keep trac responsive and at the same time keep everything
interesting in trac indexed or think about switching from sqlite to
some other db like mysql or postgres, but I would prefer to do the
first option since backup of the db is very simple for sqlite.
It seems that the crawlers have not come back since the last restart
of trac, so for now everything is good. But this issue has cost me a
couple hours getting to the bottom when I should have been merging
tickets, so I would like to solve it once and for all :)
> Carl
Cheers,
Michael
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
