Re: [Twisted-Python] Twisted webserver performance issues

2009-11-09 Thread James Y Knight
On Nov 8, 2009, at 1:08 PM, James Y Knight wrote: >> >> Can we attach gdb to the process and trigger an all-threads stack >> dump when it happens? > > That's a good idea. Unfortunately, currently it seems to be working > quite smoothly. :) The intermittent long pauses were happening again, so I

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread James Y Knight
On Nov 8, 2009, at 9:02 AM, exar...@twistedmatrix.com wrote: > On 04:26 am, f...@fuhm.net wrote: >> When I last looked into the performance issues, I found that >> sometimes >> trac appears to block for long periods of time without releasing the >> GIL. That seems to be the core of the performa

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread Nitro
Am 08.11.2009, 15:54 Uhr, schrieb Maarten ter Huurne : > On Sunday 08 November 2009, James Y Knight wrote: > >> When I last looked into the performance issues, I found that sometimes >> trac appears to block for long periods of time without releasing the >> GIL. That seems to be the core of the

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread Maarten ter Huurne
On Sunday 08 November 2009, James Y Knight wrote: > When I last looked into the performance issues, I found that sometimes > trac appears to block for long periods of time without releasing the > GIL. That seems to be the core of the performance issues, currently. > When it's responding normally,

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread exarkun
On 04:26 am, f...@fuhm.net wrote: >When I last looked into the performance issues, I found that sometimes >trac appears to block for long periods of time without releasing the >GIL. That seems to be the core of the performance issues, currently. >When it's responding normally, it's perfectly snappy

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread exarkun
On 06:26 am, ley...@ihug.co.nz wrote: >Neither of the below issues will be a problem > >Jean-Paul >Logged in users have a cookie set (I imagine). by default Varnish will >not present cached results to users with a cookie, nor will it cache >their results for others to see. So the 10% of hits (tot

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-08 Thread sstein...@gmail.com
On Nov 7, 2009, at 11:26 PM, James Y Knight wrote: > When I last looked into the performance issues, I found that sometimes > trac appears to block for long periods of time without releasing the > GIL. That seems to be the core of the performance issues, Any Earthly idea which part of trak is re

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-07 Thread Tom Leys
Neither of the below issues will be a problem Jean-Paul Logged in users have a cookie set (I imagine). by default Varnish will not present cached results to users with a cookie, nor will it cache their results for others to see. So the 10% of hits (totally my guess) that are for authenticated u

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-07 Thread James Y Knight
When I last looked into the performance issues, I found that sometimes trac appears to block for long periods of time without releasing the GIL. That seems to be the core of the performance issues, currently. When it's responding normally, it's perfectly snappy. But, sometimes, it blocks fo

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-07 Thread exarkun
On 02:55 am, ley...@ihug.co.nz wrote: >Thanks for the link info. > >I literally just set up Varnish for the first time on my web host, so I >am no pro. However, varnish uses a c like scripting language that can >run regexps on the incoming urls and apply rules to them. > >What this means is that

Re: [Twisted-Python] Twisted webserver performance issues

2009-11-07 Thread Tom Leys
Thanks for the link info. I literally just set up Varnish for the first time on my web host, so I am no pro. However, varnish uses a c like scripting language that can run regexps on the incoming urls and apply rules to them. What this means is that we can (as a starting point) choose caching

Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)

2009-11-07 Thread exarkun
On 7 Nov, 11:05 pm, exar...@twistedmatrix.com wrote: >On 10:41 pm, ley...@ihug.co.nz wrote: >> >>Isn't the simplest option to place a decent reverse proxy between the >>webserver and our clients? > >Possibly so! I gave this a half-hearted attempted a few weeks ago but >quickly gave up. If someone

Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)

2009-11-07 Thread exarkun
On 7 Nov, 11:56 pm, sstein...@gmail.com wrote: > >On Nov 7, 2009, at 6:05 PM, exar...@twistedmatrix.com wrote: >>Looking at this, I'm reminded that we should try moving the sponsor >>logos to somewhere else > >1> Move any images to CDN of any description (Cloud Files @ rackspace >does this for free

Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)

2009-11-07 Thread sstein...@gmail.com
On Nov 7, 2009, at 6:05 PM, exar...@twistedmatrix.com wrote: > Looking at this, I'm reminded that we should try moving the sponsor > logos to somewhere else 1> Move any images to CDN of any description (Cloud Files @ rackspace does this for free, if I remember correctly, included in the cost of

Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)

2009-11-07 Thread exarkun
On 10:41 pm, ley...@ihug.co.nz wrote: > >Isn't the simplest option to place a decent reverse proxy between the >webserver and our clients? Possibly so! I gave this a half-hearted attempted a few weeks ago but quickly gave up. If someone who is familiar with configuring such a proxy would like

[Twisted-Python] Twisted webserver performance issues (was volunteer...)

2009-11-07 Thread Tom Leys
Isn't the simplest option to place a decent reverse proxy between the webserver and our clients? For example, varnish will cache and proxy simple GETs, so simple views of the front page and docs (which I imagine is a large chunk of the traffic) will be offloaded from the webserver: http://v