[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
It is easy to fix things when problems are clearly reported. ;-) On Jul 29, 4:38 pm, Michael Ellis wrote: > Seems to be.  I updated, restarted, and restored the colon after the > comment.  No errors when I load the page.  Thanks for fixing it so quickly! > > On Thu, Jul 29, 2010 at 5:21 PM, mdipi

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Seems to be. I updated, restarted, and restored the colon after the comment. No errors when I load the page. Thanks for fixing it so quickly! On Thu, Jul 29, 2010 at 5:21 PM, mdipierro wrote: > Can you check if this issue is solved in trunk? > > Massimo > > On Jul 29, 7:55 am, Michael Ellis

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
Can you check if this issue is solved in trunk? Massimo On Jul 29, 7:55 am, Michael Ellis wrote: > Has anything changed at tip in the view parser? Looks like there may > be a problem now with colons at the end of comment lines. I have a > view with the following that was working fine before I up

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
good point. This runs as a daemon therefore there is no stdout by definition. I will change it into logging. On Jul 29, 11:19 am, Jonathan Lundell wrote: > On Jul 28, 2010, at 11:57 PM, mdipierro wrote: > > > I modified the expire_sessions.py in trunk and added a > > try:...execept... If the prob

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Jonathan Lundell
On Jul 28, 2010, at 11:57 PM, mdipierro wrote: > I modified the expire_sessions.py in trunk and added a > try:...execept... If the problem was that a file was locked and the > cron task got stuck, it may be solved now. Can you test it? In this patch: @@ -6,8 +6,14 @@ now=time.time() for file i

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
without On Thu, Jul 29, 2010 at 11:38 AM, mdipierro wrote: > with or without -N > > On Jul 29, 10:27 am, Michael Ellis wrote: > > No process leakage in instances I've been running since this morning. > Open > > file count appears stable, too. > > Cheers, > > Mike > > > > On Thu, Jul 29, 2010

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
with or without -N On Jul 29, 10:27 am, Michael Ellis wrote: > No process leakage in instances I've been running since this morning.  Open > file count appears stable, too. > Cheers, > Mike > > On Thu, Jul 29, 2010 at 9:22 AM, mdipierro wrote: > > we have a new template parser. My guess is that

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
No process leakage in instances I've been running since this morning. Open file count appears stable, too. Cheers, Mike On Thu, Jul 29, 2010 at 9:22 AM, mdipierro wrote: > we have a new template parser. My guess is that it is not ignoring > the : at the end of {{# comments. > > On Jul 29, 7:55

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
we have a new template parser. My guess is that it is not ignoring the : at the end of {{# comments. On Jul 29, 7:55 am, Michael Ellis wrote: > Has anything changed at tip in the view parser? Looks like there may > be a problem now with colons at the end of comment lines. I have a > view with the

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Has anything changed at tip in the view parser? Looks like there may be a problem now with colons at the end of comment lines. I have a view with the following that was working fine before I updated this morning. {{ ## On entry, vars returned by controller are: ## problem, c, s, sc, wsc, problemid

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Testing under way... (Iceberg, I'm running OS X 10.6.4) Cheers, Mike On Thu, Jul 29, 2010 at 2:57 AM, mdipierro wrote: > Mike, > > I modified the expire_sessions.py in trunk and added a > try:...execept... If the problem was that a file was locked and the > cron task got stuck, it may be solv

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
Mike, I modified the expire_sessions.py in trunk and added a try:...execept... If the problem was that a file was locked and the cron task got stuck, it may be solved now. Can you test it? Massimo On Jul 28, 4:10 pm, MikeEllis wrote: > Update: (Summarizes a couple of emails exchanged with Massi

[web2py] Re: rocket performance issues

2010-07-28 Thread Iceberg
The one "leak" process per hour coincident is "coincidently" matching the one run per hour of admin/cron/expire_sessions.py That really raises my concern, cause I just gonna use cron as a trick to touch my app's cache frequently. :-/ By the way, which os is Mike Ellis using? I did not observe sam

[web2py] Re: rocket performance issues

2010-07-28 Thread mdipierro
There clearly is a problem with the cron processes. Massimo On Jul 28, 4:10 pm, MikeEllis wrote: > Update: (Summarizes a couple of emails exchanged with Massimo) > > Yesterday I found that my instance of web2py appeared be "leaking" > processes at about 1 per hour.  Massimo suggested running wit

[web2py] Re: rocket performance issues

2010-07-28 Thread MikeEllis
Update: (Summarizes a couple of emails exchanged with Massimo) Yesterday I found that my instance of web2py appeared be "leaking" processes at about 1 per hour. Massimo suggested running with -N to disable cron. I started two instances this way -- one under winpdb and another directly. They've

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Jonathan Lundell
On Jul 27, 2010, at 9:01 AM, Michael Ellis wrote: > Thanks for the info about sloth. I may try it later. It was pretty easy to > get what I needed with ipython: > > procs = !ps ax | grep "web2py" > pids = [r[0] for r in procs.fields()[0:-2]] > for p in pids: > print p > !lsof | grep $

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Sorry, what I sent was very large (~750K each). Didn't realize GoogleGroups would reduce it to a thumbnail. I'll try to forward the original message directly to you ... On Tue, Jul 27, 2010 at 12:07 PM, mdipierro wrote: > The picture is too small. Anything interesting in there? > > On Jul 27

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
The picture is too small. Anything interesting in there? On Jul 27, 11:01 am, Michael Ellis wrote: > Thanks for the info about sloth. I may try it later.  It was pretty easy to > get what I needed with ipython: > > procs = !ps ax | grep "web2py" > pids = [r[0] for r in procs.fields()[0:-2]] > for

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Thanks, Jonathan. That's a good suggestion. I last restarted the app a couple of hours ago. ps reports 4 processes that match 'web2py'. Grepping for each pid in lsof shows nothing extraordinary. Each process has 80-90 files open. The vast majority are libraries (.so). There are a few chr files

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Jonathan Lundell
On Jul 27, 2010, at 5:33 AM, Michael Ellis wrote: > This morning I found several OSError reports about "Too many open files" in a > web2py development server that's been running locally for several days. The > app code isn't doing any explicit file i/o so I don't know what's going on, > but h

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
ok, I'll give that a shot in a few minutes if I don't get a fork break by then. BTW, I verified that cron/crontab is empty in my app. However it's not empty in admin and examples: $ cat admin/cron/crontab 10 * * * * root **applications/admin/cron/expire_sessions.py $ cat examples/cron/crontab

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Yet is being called. It is on by default. Try set cron=False at the bottom of web2py.py On Jul 27, 7:49 am, Michael Ellis wrote: > Not using cron. > > On Tue, Jul 27, 2010 at 8:43 AM, mdipierro wrote: > > Ignore my previous email I see you are using 2.6 and the problem > > is with cron. I th

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
I think I can test the cron hypothesis. I'm running web2py under winpdb. Normally, I set the fork mode to "parent, auto" so that it doesn't break when a child process starts. If I change it to manual, it should break next time a child process is spawned. What should I look for when that happens?

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Not using cron. On Tue, Jul 27, 2010 at 8:43 AM, mdipierro wrote: > Ignore my previous email I see you are using 2.6 and the problem > is with cron. I think the problem is a cron process that does not end > and keeps restarting. Are you using cron? > > On Jul 27, 7:33 am, Michael Ellis w

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Ignore my previous email I see you are using 2.6 and the problem is with cron. I think the problem is a cron process that does not end and keeps restarting. Are you using cron? On Jul 27, 7:33 am, Michael Ellis wrote: > Not sure if this is related to Rocket or whether a new topic is needed. >

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Are you opening any files explicitely in your app? Which web2py version and which Python version are you using? On Jul 27, 7:33 am, Michael Ellis wrote: > Not sure if this is related to Rocket or whether a new topic is needed. > > This morning I found several OSError reports about "Too many open

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Not sure if this is related to Rocket or whether a new topic is needed. This morning I found several OSError reports about "Too many open files" in a web2py development server that's been running locally for several days. The app code isn't doing any explicit file i/o so I don't know what's going

[web2py] Re: rocket performance issues

2010-07-26 Thread Rahul
Thanks everyone, for getting this issue resolved... Web2py rocks! Cheers, Rahul On Jul 24, 6:25 pm, Phyo Arkar wrote: > Yes , Since from start of web2py , Massimo and Us never recommend to use > built-in web2py server at production. They are mainly for development > purpose. > Since Very start of

Re: [web2py] Re: rocket performance issues

2010-07-24 Thread Phyo Arkar
Yes , Since from start of web2py , Massimo and Us never recommend to use built-in web2py server at production. They are mainly for development purpose. Since Very start of the project , since Cherrypy , Massimo always Suggest to use Apache/Cherokee/LighHTTPD over fcgi/wsgi/uwsgi or mod_python in Se

[web2py] Re: rocket performance issues

2010-07-24 Thread Scott
Please allow me to preface my comments: I have nothing against Rocket; my opinions come from years of experience with Java EE deployments. I think raising the max_threads to 1024 is a good idea. However, my opinion is that Rocket alone should not be used for a production deployment; much as I wou

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
On a second thought this open the door to more sever denial of service attacks than caused by the original problem. How about, until there is a better under understanding and solution, we just increase max_threads from the original 128 to 1024. On Jul 22, 11:27 am, Timbo wrote: > Try one quick ch

Re: [web2py] Re: rocket performance issues

2010-07-23 Thread Michele Comitini
Yes that could be really a small problem with Rocket's threaded implementation, not the GIL, which usually shows up on higher concurrency loads. Anyway that would be a point for Guido... :D ciao, mic 2010/7/23 mdipierro > I am not convinced this is a GIL problem. Why the lock after the 400 >

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
I am not convinced this is a GIL problem. Why the lock after the 400 errors only? On Jul 23, 9:18 am, Michele Comitini wrote: > Because of the GIL threads on CPython are problematic more than one could > think.  Googling "GIL python" gives an idea... > Not to start the war threading vs forking mo

Re: [web2py] Re: rocket performance issues

2010-07-23 Thread Michele Comitini
Because of the GIL threads on CPython are problematic more than one could think. Googling "GIL python" gives an idea... Not to start the war threading vs forking model in general, but the GIL will not go away from van Rossum's python implementation (no not even in 3.x), he rather suggests to stay

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
Good idea. In trunk. On Jul 23, 6:47 am, Iceberg wrote: > Nice to have partial progress in 1.81.5 anyway. Congratulations. Even > so, how comes an HTTP 400 error can lock Rocket? �...@_@ > > And if that is somehow really the cause, we need to prevent 400 error > from /favicon.ico  It is simple, a

[web2py] Re: rocket performance issues

2010-07-23 Thread Iceberg
Nice to have partial progress in 1.81.5 anyway. Congratulations. Even so, how comes an HTTP 400 error can lock Rocket? @_@ And if that is somehow really the cause, we need to prevent 400 error from /favicon.ico It is simple, add this lines into welcome scaffold's layout.html and supply a def

[web2py] Re: rocket performance issues

2010-07-22 Thread mdipierro
This is now in 1.81.5 On Jul 22, 11:18 pm, Phyo Arkar wrote: > Thats very good news. I will be trying out soon! > > On 7/22/10, Timbo wrote: > > > OK, so it's a thread-starvation issue.  Let this be the interim fix > > for web2py.  I've got several busy days ahead of me so I'll put out a > > rea

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Phyo Arkar
Thats very good news. I will be trying out soon! On 7/22/10, Timbo wrote: > OK, so it's a thread-starvation issue. Let this be the interim fix > for web2py. I've got several busy days ahead of me so I'll put out a > real fix for Rocket in the coming days. > > -tim > > On Jul 22, 11:39 am, Micha

[web2py] Re: rocket performance issues

2010-07-22 Thread Timbo
OK, so it's a thread-starvation issue. Let this be the interim fix for web2py. I've got several busy days ahead of me so I'll put out a real fix for Rocket in the coming days. -tim On Jul 22, 11:39 am, Michael Ellis wrote: > Much improved with suggested change.  Latencies for .js/.css files re

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Michael Ellis
Much improved with suggested change. Latencies for .js/.css files reduced to 100-150 msec. Thanks Tim! Cheers, Mike On Thu, Jul 22, 2010 at 12:27 PM, Timbo wrote: > Try one quick change for me please...rocket is constructed around line > 655 in main.py > > Add a parameter to the constructor

[web2py] Re: rocket performance issues

2010-07-22 Thread Timbo
Try one quick change for me please...rocket is constructed around line 655 in main.py Add a parameter to the constructor call(s): max_threads=0 Please let me know if that affects the problem. -tim On Jul 22, 10:34 am, mdipierro wrote: > I can reproduce the problem. I did on localhost with two

[web2py] Re: rocket performance issues

2010-07-22 Thread mdipierro
I can reproduce the problem. I did on localhost with two different browsers. Using firebug I can see it takes 25seconds to download base.css (the problem is not always with the same file). While I did the test, I also monitored httpserver.log and I find that it NEVER takes more than 1.2ms serve bas

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Jonathan Lundell
On Jul 22, 2010, at 6:55 AM, MikeEllis wrote: > Doesn't fix the symptom Ah. Well, negative results are still results. > > On Jul 21, 8:54 pm, Jonathan Lundell wrote: >> On Jul 21, 2010, at 2:03 PM, mdipierro wrote: >> >>> The fact that you are having the problem with static files is giving >

[web2py] Re: rocket performance issues

2010-07-22 Thread MikeEllis
Doesn't fix the symptom On Jul 21, 8:54 pm, Jonathan Lundell wrote: > On Jul 21, 2010, at 2:03 PM, mdipierro wrote: > > > The fact that you are having the problem with static files is giving > > me an idea for a test. > > Can you please try the web2py I just posted in trunk? > > That seems like a

Re: [web2py] Re: rocket performance issues

2010-07-21 Thread Jonathan Lundell
On Jul 21, 2010, at 2:03 PM, mdipierro wrote: > The fact that you are having the problem with static files is giving > me an idea for a test. > Can you please try the web2py I just posted in trunk? That seems like a nice optimization, but (assuming it fixes the symptom) does it explain the 20-25

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
Alas, no change with Version 1.81.4 (2010-07-17 16:14:44) One more little observation: In general, I can load a page from either, but not both, of the browsers repeatedly and not get the problem. It happens with alternation, as discussed previously, but I now notice that I seem to be able to ca

[web2py] Re: rocket performance issues

2010-07-21 Thread mdipierro
The fact that you are having the problem with static files is giving me an idea for a test. Can you please try the web2py I just posted in trunk? Massimo On Jul 21, 2:08 pm, MikeEllis wrote: > One more detail.  As you would hope, it's generally not the case that > both browsers stall at the same

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
One more detail. As you would hope, it's generally not the case that both browsers stall at the same time. While stalled on either laptop I can keep reloading on the other. On Jul 21, 3:03 pm, MikeEllis wrote: > I tried changing to cron=False in web2py.py.  No impact on problem. > Same two-lapt

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
I tried changing to cron=False in web2py.py. No impact on problem. Same two-laptop setup as described previous. Happens even on pages that contain very little. Chrome DevTools shows same problem as before: One or more the .js/.css/.png files are delayed by 20 to 25 seconds latency. The proble

Re: [web2py] Re: rocket performance issues

2010-07-21 Thread Kuba Kucharski
yes, but I am on the GPRS connection now, I am not able to test it so the results be reliable, I will try to do that asap

[web2py] Re: rocket performance issues

2010-07-21 Thread mdipierro
Kuba, please try set cron=False in web2py.py. Does it make things better? Massimo On Jul 21, 10:33 am, Kuba Kucharski wrote: > "sometimes, once every few hits, page loads significantly slower > weirdest thing is when you re-click the link it loads instantly, when > you left it working to load on