I appreciate you sharing your knowledge on this! Sounds like it may have been a bad commit? Or, at least there should be some mention in web2py documentation about changing these values, mainly based on hardware?
Thanks again, sounds like my best bet is to reconfigure the server for quad core CPU, and set processes=4, and leave threads=15 (or lower, but not 1) Dave On Friday, 31 July 2015 00:24:32 UTC-6, Niphlod wrote: > > let's not spread misinformations .... python has the GIL, true, and 15 > threads with a single process won't use any cpu a single thread can use, > nor span multiple cpus. But for anyone's sake, we're deploying a webapp in > PRODUCTION, whose purpose should be - at the very least - be concurrent. > If we force 1 process and a single thread, make ourselves a clap, than > bang the head to the wall. we're serializing everything! > > tl;dr: I understand someone may think that 1 process with 15 threads > (defaults of the apache directive) slow things down when you have 4xcpu and > 3 of them remain underutilized while serving concurrently 15 requests (buh > huh GIL, but I'd like to see test results), and that maybe a 4 processes > with each 5 threads would be a smarter choice (really?!) but 1 process and > 1 thread is a complete joke. > > On Friday, July 31, 2015 at 8:18:23 AM UTC+2, Derek wrote: >> >> I believe that's what Massimo is trying to say. Each processor has it's >> own GIL. >> >> On Thursday, July 30, 2015 at 6:31:57 PM UTC-7, Dave wrote: >>> >>> Thanks for tracking that down! So based on that post, and this one, it >>> looks like i should be setting proccesses = # of cores, and threads = 1? >>> >>> https://groups.google.com/forum/#!topic/web2py/mPdn1ClxLTI >>> >>> Massimo DI Pierro: >>> There are pros and cons. If you use threads in a python program, the >>> more computing cores you have, the slower - not faster - the program gets. >>> This is a python feature because even if you have threads, there is only >>> one interpreted and therefore execution is serialized anyway. For >>> scalability you should have processes (not threads) one per core. >>> >>> >>> >>> On Thursday, 30 July 2015 14:15:36 UTC-6, Derek wrote: >>>> >>>> looks like that change came from here: >>>> >>>> >>>> https://groups.google.com/forum/#!searchin/web2py/processes$3D1$20threads$3D1|sort:date/web2py/jumZFKX2614/pu-NNXSMKHgJ >>>> >>>> based on the post from Thomas... >>>> >>>> Thomas J. >>>> 10/3/13 >>>> I've recently been comparing Web2py and PHP, this is what i found, >>>> maybe it helps: >>>> >>>> 1.: PHP is faster whereever it can do stuff within the interpreter, >>>> that includes handling post/get-data or templates. The PHP interpreter is >>>> written in C, so these things are really fast, whereas Web2py has to to >>>> them in Python. >>>> 2. DB access is heavily dependent on how many records you retrieve. >>>> Translating a query from DAL to SQL is basically free (so using the DAL >>>> syntax for DB access isn't an issue), putting the data into Python objects >>>> is quite expensive however. Only query what you really need. Also, using >>>> executesql() instead of the regular DAL syntax may help there, as it >>>> returns tuples, not complex data structures >>>> 3. I've found the default config for Apache2 to be somewhat broken as >>>> far as concurrency is concerned. Check your Apache/WSGI config, you >>>> probably have a line like "WSGIDaemonProcess web2py user=www-data >>>> group=www-data" in there. This actually defaults to 1 process with 15 >>>> threads, which -- on my machine -- completely kills performance (Python >>>> doesn't do threads well because of the global interpreter lock). I've >>>> found >>>> that even something like "WSGIDaemonProcess web2py user=www-data >>>> group=www-data processes=1 threads=1" improves performance for concurrent >>>> requests dramatically. Try tinkering with the processes-value to find the >>>> best config for your machine. >>>> >>>> You should probably change processes before you change threads. >>>> >>>> On Wednesday, July 29, 2015 at 3:40:30 PM UTC-7, Dave wrote: >>>>> >>>>> That would be good to know. Nothing in the web2py docs suggests that >>>>> Apache should not be used in production, it actually seems more like the >>>>> default since it is the first discussed in the documentation and it's >>>>> used >>>>> in the one-step production deployment. >>>>> >>>>> The line of code in question is also in the one-step deployment script >>>>> here: >>>>> http://web2py.googlecode.com/hg/scripts/setup-web2py-ubuntu.sh >>>>> >>>>> >>>>> On Wednesday, 29 July 2015 14:32:19 UTC-6, Niphlod wrote: >>>>>> >>>>>> I dropped off the apache train too soon to have any issues with it, >>>>>> but frankly, given the total sum of issues encountered so far on the >>>>>> forums, I'm starting to think that we'd need to "officially discontinue" >>>>>> our apache support.. may be total lack of luck in setting it up or very >>>>>> biased perspective, or total lack of internal knowledge but it seems >>>>>> that >>>>>> every problem that pops up with deployments have apache as the common >>>>>> ground. >>>>>> >>>>>> looks like this is the commit to be blamed >>>>>> >>>>>> >>>>>> https://github.com/web2py/web2py/commit/2a062a2ff5aa1e07e7bfcfdbf36b7f72e8aac5b4 >>>>>> >>>>>> I don't know the specifics around it but if it acts like it suggests, >>>>>> 1 thread and 1 process as a total sum aren't really worth of a >>>>>> production >>>>>> deployment. >>>>>> >>>>>> >>>>>> On Wednesday, July 29, 2015 at 6:00:47 PM UTC+2, Dave wrote: >>>>>>> >>>>>>> Actually, it looks like i was chasing the wrong issue... It wasn't >>>>>>> https after all. >>>>>>> >>>>>>> Everything seems to be working after changing this line in apache >>>>>>> default.conf: >>>>>>> >>>>>>> WSGIDaemonProcess web2py user=www-data group=www-data processes=1 >>>>>>> threads=1 >>>>>>> >>>>>>> >>>>>>> to: >>>>>>> >>>>>>> WSGIDaemonProcess web2py user=www-data group=www-data processes=5 >>>>>>> threads=15 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Is there any reason not to change this default setting from one-step >>>>>>> deployment? Can I likely set these values higher based on my hardware? >>>>>>> >>>>>>> Thanks again, >>>>>>> Dave >>>>>>> >>>>>>> On Wednesday, 29 July 2015 02:52:20 UTC-6, Niphlod wrote: >>>>>>>> >>>>>>>> uhm, you left out some pretty specific details.... what resources >>>>>>>> has the server web2py is deployed on ? moreover, what's the size of >>>>>>>> the >>>>>>>> file ? and what code are you using to handle the upload? are you using >>>>>>>> the >>>>>>>> default 'upload' Field or is it in conjunction with a 'blob' one to >>>>>>>> store >>>>>>>> the file on the database ? >>>>>>>> >>>>>>>> On Wednesday, July 29, 2015 at 4:59:29 AM UTC+2, Dave wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> I have this same behavior on multiple web2py servers. If a large >>>>>>>>> file is being uploaded using a SQLFORM or downloaded using the >>>>>>>>> default >>>>>>>>> download controller, over HTTPS, the entire web server becomes >>>>>>>>> unresponsive >>>>>>>>> until the transfer is completed or cancelled. However, I have no >>>>>>>>> issues >>>>>>>>> uploading/downloading the same file over HTTP, which can also take >>>>>>>>> several >>>>>>>>> minutes to complete, but the web server is still responsive during >>>>>>>>> those >>>>>>>>> transfers. >>>>>>>>> >>>>>>>>> I am using the one-step deployment with Apache and a wildcard >>>>>>>>> certificate (RapidSSL). Would switching to nginx or cherokee give >>>>>>>>> better >>>>>>>>> performance for https file transfers, or is this likely an issue with >>>>>>>>> the >>>>>>>>> SSL certificate format? Or if the file transfers over HTTPS are too >>>>>>>>> CPU >>>>>>>>> intensive, am i better off setting up multiple servers and a load >>>>>>>>> balancer? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> Dave R >>>>>>>>> >>>>>>>>> >>>>>>>>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
