What I mean is ferwer requests than all requests actually change the session.
On Mar 14, 11:58 pm, VP <vtp2...@gmail.com> wrote: > I think when I speak of scalability we shouldn't make the assumption > that "requests [are coming from] only a few users". That said, it's > great to know session handling is being optimized. I think smart > session handling and smart cache handling will greatly improve > performance. > > On Mar 14, 11:19 pm, Massimo Di Pierro <massimo.dipie...@gmail.com> > wrote: > > > > > > > > > This also explains why the problem occurred with Apache ab but not for > > (usually) people using web2py in production. In production you do not > > have thousand of session files because many of the requests from from > > the same few users. > > > There are exceptions. Apps that have many semi-static pages or are > > probed by ab or are subject to DoD attacks. those apps may have > > suffered from the problem that is now fixed. > > > Massimo > > > On Mar 14, 11:12 pm, Massimo Di Pierro <massimo.dipie...@gmail.com> > > wrote: > > > > I think also address the scalability problem that has occasionally > > > been reported using ab tests. > > > > Apache ab sends many uncorrelated requests and each of them results > > > (resulted) in a new session file saved. Eventually the more tests you > > > run the more files you have. The more files in the same folder you > > > have, the slower the file system becomes, the longer it takes web2py > > > to save a session, the longer the thread runs, the less requests/ > > > second you can process, the sooner you reach the max number of threads > > > and requests will be dropped. > > > > The latest fix should address the issue since only REAL sessions will > > > not result in file system access. > > > > It should also considerably speed up GAE since less datastore write > > > access. > > > > There may be room for further improvement but I believe this is a big > > > improvement already. > > > > Please report any data you may have. > > > > Massimo > > > > On Mar 14, 10:40 pm, Massimo Di Pierro <massimo.dipie...@gmail.com> > > > wrote: > > > > > At PyCon some people have suggested that they would like the ability > > > > to switch off some web2py functionality to make it faster. I never > > > > thought of this being an issue because I do not think web2py does much > > > > outside models/views and controllers. > > > > > I run a lot of tests today using a barebone > > > > > def index(): return 'hello world' > > > > > controller (no view, no model) and I discovered, to my surprise that > > > > even in this simple case, web2py spends huge amount of time saving > > > > sessions even if not used (~40ms/request on my laptop). This is also a > > > > problem because if you have many passive visitors web2py creates a lot > > > > of empty session files that fills the sessions folder and slow down > > > > the file system even more. > > > > > I made some changes so that sessions are no longer saved if not > > > > modified or empty. For my simple code, the running time went down to > > > > 4ms/request. That is 10x than before. That is only 5x the bare Rocket > > > > or Tornado speed (considering it is still checking for session, doing > > > > routing, parsing input and performing a lot of checks, and a lot of > > > > other stuff). > > > > > I do not think it should break anything and you should feel it > > > > snappier immediately. Give it a try and let me know. > > > > > Try run ab -n 1000 with both web2py 1.93.1 and trunk for your app. Let > > > > me know what you get. > > > > > Enjoy! > > > > > Massimo
