Hi All, Thanks for the enlightening discussion. web2py community is awesome!
I am hoping to create a rudimentary implementation and perform load testing etc. My plan is to spread out over multiple machines (2 for a start), say end-Aug. I am planning to use apache+mod_wsgi I will share the lessons I learn on this forum and this wiki page: https://mdp.cti.depaul.edu/wiki/default/page/8692b276-3ddf-4038-9525-bc51060ad1dc Regards Anand On Jul 22, 5:23 pm, Alex Fanjul <[email protected]> wrote: > Thanks Massimo, Bottiger, good to know some keys that I didn't know, > like tricks to speed up the app, and the python efficiency in multicore. > Excuse my ignorance , but let me enquire a bit more... > > * 10x more requests by only installing a simple package (flup)? or > is an special server? or is superman? > o Bottiger, I tried to find out some information and > documentation about flup but -apart from official trac wiki- > it's lack of > o Could you give us some introduction about Flup, Prefork > server (or whatever it is) and how to install it to improve > the performance? > * So (according to what I've been reading) I conclude that a great > performance machine for stressed production enviroment would be > made up of: > o Multiple core machine + > o Flup Server packages + > o Cherokee + > o Postgres + > o Web2py Tricks > * Massimo, in a old thread I read something about a prebuilt vmware > with cherokee and web2py, ready to install, is it any news about this? > o In terms of "marketing", I think that if we could upload a > vmware turnkey appliance to here > <http://www.vmware.com/appliances/> (like Django one > <http://www.vmware.com/appliances/directory/82433>), it > would be increase our presence > > Thanks, > alex f > > El 22/07/2009 10:48, mdipierro escribió: > > > > > With 8 cores and Flup you can probably get close to 1000 requests/ > > second. > > if you can do any test, let us know what you get. > > > Massimo > > > On Jul 22, 3:22 am, Bottiger<[email protected]> wrote: > > >> "There are some things that definitively will NOT help: a multicore > >> machine. because the python interpreter cannot use > >> multiple cores efficiently " > > >> If you use the prefork flup server not included in Web2Py, each > >> request is handled by a seperate process so this will bypass the GIL. > >> It makes a huge difference on my 8 core server. Presumably, Jython > >> which is GILless would yield the same performance on a multicore > >> machine. > > >> "I will actually argue most of the map/reduce DB out there are slower > >> then postresql. " > > >> For minimal latency, this is probably true, but I am willing to bet > >> that the latency grows far slower in a map/reduce DB as the number of > >> simultaneous connection rises. > > >> On Jul 22, 1:03 am, mdipierro<[email protected]> wrote: > > >>> This is a complex issues that spans different topics: speed, > >>> efficiency, scalability. > > >>> I am not aware of any major bottle neck in web2py, execpt of the > >>> database (not the DAL , the actual database) so it is efficiently. > >>> There are many little tricks you can use to speed applications more: > >>> - use connection pooling > >>> - run your app bytecode compiled (press the button) > >>> - move as much code as you can modules instead of models. > >>> - discard sessions unless you they have modified > >>> - store sessions in a memory mapped file > > >>> How fast is it (with or without optimizations) depends on the > >>> architecture. Is there a machine that can give you 1000 request/ > >>> seconds. I do know. It is possible. On my virtual machine I get about > >>> 100. > > >>> There are some things that definitively will NOT help: > >>> - a multicore machine. because the python interpreter cannot use > >>> multiple cores efficiently > >>> - a key/value database. This helps with scalability (i.e. run lots of > >>> concurrent servers) but not necessarily speeds up a single server. I > >>> will actually argue most of the map/reduce DB out there are slower > >>> then postresql. > > >>> Massimo > > >>> On Jul 21, 8:40 pm, Alex Fanjul<[email protected]> wrote: > > >>>> Hello Massimo (all) this days I'm reading about horizontal scale > >>>> architectures, key/value and graphs db's, etc. and the awakening in > >>>> cloud computing enviroment > >>>> In the last reading, I saw "Redit" benchmark of about 50 to 100 > >>>> thousands req/sec with standard linux box. > > >>>> I know those values are due to DB architecture of key/value (and they > >>>> are really incredible), but: > >>>> -what thing is really limiting 1000 req/sec in web2py? > >>>> cherrypy/apache? mysql/postgres? wscgi/fastcgi? web2pyframework? python? > >>>> -what do you think would the upper limit (req/sec) be in the better > >>>> production enviroment case (great linux server/s, apache/cherokee?, best > >>>> connection) > >>>> -As a matter of curiosity, have you ever though in implementing the > >>>> API for any of such databases? Redit? Tokyo? couchDB? > > >>>> regards, > >>>> alex f > > >>>> P.S as always i'm sorry for my poor english > > >>>> El 21/07/2009 15:52, mdipierro escribió: > > >>>>> - Are there any large web2py installations that I can quote as an > >>>>> example > > >>>>> Not that I know and handle>1000 requests/second. > > >>>>> - How are the issues of caching (say rendered pages) handled? I have > >>>>> done a few Drupal sites and can see the performance effects of caching > >>>>> very clearly. IIRC only Django has caching in the python world? > > >>>>> If you use multiple installations behind a load balancer I suggest you > >>>>> use the "pound" load balancer to keep sessions sticky. In that case > >>>>> the different processes do not need to share any data. > > >>>>> - Has anyone done any work with web2py in a cluster (similar to a > >>>>> Tomcat cluster behind mod_jk)? (multiple machines running web2py, the > >>>>> session data sync'd etc. I can put the session info in a shared FS > >>>>> though) > > >>>>> If you need sessions and you need sessions synced, I suggest you share > >>>>> the sessions folder. > > >>>>> Massimo > > >>>>> On Jul 21, 12:20 am, Bottiger<[email protected]> wrote: > > >>>>>> If it is truly not computationally intensive, and does not even use a > >>>>>> database, it should not be a problem. > > >>>>>> I have benchmarked Web2Py on the static welcome page to 700 requests/ > >>>>>> second with a concurrency level of 50. > > >>>>>> To increase the level of concurrency (if you have additional CPU > >>>>>> cores), you should increase the number of Web2Py processes. > > >>>>>> "~ 8000 users, and atleast 1000-2000 simultaneous users." > > >>>>>> This is not really a large installation if it doesn't use a database. > > >>>>>> "How are the issues of caching (say rendered pages) handled? I have > >>>>>> done a few Drupal sites and can see the performance effects of caching > >>>>>> very clearly. IIRC only Django has caching in the python world?" > > >>>>>> Drupal, Django, and Web2Py have equivalent caching mechanisms. Any > >>>>>> external caching mechanism you have seen with Drupal should also be > >>>>>> usable with Web2Py or Django. > > >>>>>> "I can put the session info in a shared FS though" > > >>>>>> You can either do that or use a database for sessions. > > >>>>>> On Jul 20, 8:54 pm, Anand Vaidya<[email protected]> wrote: > > >>>>>>> Hi > > >>>>>>> After a couple of web2py projects, I am confident of coding a fairly > >>>>>>> big app in web2py. > > >>>>>>> My previous projects did not need any database (we had to use > >>>>>>> flatfiles), the new project is also similar. I intend to bypass the > >>>>>>> models etc completely. > > >>>>>>> The app is likely to be used in a corporate setting with ~ 8000 users, > >>>>>>> and atleast 1000-2000 simultaneous users. > > >>>>>>> The users authenticate to an LDAP server. > > >>>>>>> The app is not computationally intensive > > >>>>>>> It queries another service and displays results > > >>>>>>> No SQL DB is required > > >>>>>>> Most likely behind a few Apache 2.x front server > > >>>>>>> I'd like to know: > > >>>>>>> - Are there any large web2py installations that I can quote as an > >>>>>>> example > > >>>>>>> - How are the issues of caching (say rendered pages) handled? I have > >>>>>>> done a few Drupal sites and can see the performance effects of caching > >>>>>>> very clearly. IIRC only Django has caching in the python world? > > >>>>>>> - Has anyone done any work with web2py in a cluster (similar to a > >>>>>>> Tomcat cluster behind mod_jk)? (multiple machines running web2py, the > >>>>>>> session data sync'd etc. I can put the session info in a shared FS > >>>>>>> though) > > >>>>>>> Regards > >>>>>>> Anand > > >>>> -- > >>>> Alejandro Fanjul Fdez. > >>>> [email protected] > > -- > Alejandro Fanjul Fdez. > [email protected] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
