Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-06 Thread Lisandro Rostagno
Thank you very much Michele for your answer. I understand that assigning different priorities to each web app is a complex problem. I mean, it's "complex" in the sense that it involves many different areas. But I think that with the information I have and the configuration variables I know, I alrea

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-05 Thread Michele Comitini
I've made this change to one of the apps. I'll be testing with other > apps in the next days. To get it running I had to import "fcgi_fork" > from flup.server instead of simple "fcgi". > > I know this subject is way away from this group, but I'm tempted to > ask, given the quality of the info I've

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-04 Thread Lisandro Rostagno
2014-12-03 14:16 GMT-03:00 Michele Comitini : > why not flup? > - not event based. Event based servers are usually more responsive under > load. > - seems pretty unmantained. > - misses some tuning options. I see. I've read a little about gevent and I understand that it would be a valuable incorp

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-03 Thread Michele Comitini
why not flup? - not event based. Event based servers are usually more responsive under load. - seems pretty unmantained. - misses some tuning options. max-procs in lighttpd configuration: - leave it to 1 - work with the following options in your script: WSGIServer(application, minSpare=1, maxSpar

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-03 Thread Lisandro Rostagno
Actually I was referring to max-procs parameter of the fastcgi server that is setup in lighttpd virtual host configuration. I don't know if you mean that. In my virtual host configuration, I setup the fastcgi server like this: fastcgi.server = ( ".fcgi" => ("localhost" => (

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-02 Thread Michele Comitini
Everything ok with your plan. One note, instead of playing with the max-procs in lighttpd you can use the fastcgi server parameters. btw don't use flup use this: https://github.com/momyc/gevent-fastcgi and arrange the num_workers parameter. to do that differently for each application may end up wi

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-01 Thread Lisandro Rostagno
Thank you very much Niphlod and Michele for your help! Now I have a more clear understanding of all this. I understand now why I was seeing 5 fastcgi processes regardless of the "max-procs" configuration (that was because I was using fcgi_fork, and I was actually seeing one process and its 5 childr

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-01 Thread Michele Comitini
What you have here is lighttpd starting 1 forking flup server. The important part is this: from flup.server.fcgi_fork import WSGIServer With that you have a python interpreter for each request, up to a maximum. The default is 5 spare children, the default maximum is 50. Under lighttpd is also p

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-01 Thread Niphlod
@lisandro: michele is right.. the pool_size parameter calculation are accurate only if there's one process per app. web2py can't coordinate pools among different processes... also, max_client_conn is exactly the maximum number of connection the pgbouncer process will allow "coming in". Once over

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-12-01 Thread Lisandro Rostagno
Sorry about the delay. I recently installed pgbouncer. I let postgresql max_connection set to 80, and configure pgbouncer with a max_client_conn of 1000 and a default_pool_size of 20. Now when I check pg_stat_activity I can see different amounts of idle connections per app, more accordingly with th

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-11-30 Thread Michele Comitini
p.s. by "no threading" I mean to use processes in place of threads. The number of processes is something you must tune based on server resources, 2xn where n is the number of cores is a safe choice. 2014-11-30 20:04 GMT+01:00 Michele Comitini : > pool_size==number of threads in a web2py process.

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-11-30 Thread Michele Comitini
pool_size==number of threads in a web2py process. I suggest to work around the problem by setting the number of threads to 1 in you flup server. I.e. no threading. You should also see smoother performance across applications on higher loads. 2014-11-30 15:51 GMT+01:00 Lisandro Rostagno : > Yes

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-11-30 Thread Lisandro Rostagno
Yes in deed. I've restarted the webserver and the database server. Recently I've tried setting pool_size to 1 for every app, that is, for every website. Restarted postgresql and webserver (lighttpd). And then I used this SQL statement to check the total count of connections for every database (or w

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-11-30 Thread Niphlod
did you restart the webserver ? I don't think that changing pool_size at runtime when connections are still open will make the number of active connection dropped. On Friday, November 28, 2014 8:48:07 PM UTC+1, Lisandro wrote: > > Mmm... I see. That was my understanding in the first place. > At

Re: [web2py] Re: Web2py's included connection pooling VS external pooling software

2014-11-28 Thread Lisandro Rostagno
Mmm... I see. That was my understanding in the first place. At that time I did the maths, I had 10 apps, each one using a pool_size of 3. In postgresql.conf max_connections was set to 80. However this morning, with those numbers, almost every of my websites was throwing intermitent HTTP 500 errors,