[web2py] Re: IOError, scalability concerned

2010-05-10 Thread mdipierro
A smarter approach could be not generate a ticket if the socket is closed but I think still Graham objection would still be valid. On May 10, 6:51 pm, Thadeus Burgess wrote: > There are alot of these tickets being generated, roughly 10%, this is > annoying from a prospective of attempting to dete

Re: [web2py] Re: IOError, scalability concerned

2010-05-10 Thread Thadeus Burgess
There are alot of these tickets being generated, roughly 10%, this is annoying from a prospective of attempting to determine the tickets that are real! Let alone the amount of files it is placing in the errors folder quickly exceeds the recommended max file count for a folder. I wish it would at l

[web2py] Re: IOError, scalability concerned

2010-05-10 Thread mdipierro
Good point. I guess I would revert the latest change to wsgibase() let me me think about this some more. On May 10, 5:48 pm, Graham Dumpleton wrote: > On May 10, 3:01 pm, mdipierro wrote: > > > Graham is right. The best we can do is catch > > > try: > >    data = src.read(size) > > except: > >  

[web2py] Re: IOError, scalability concerned

2010-05-10 Thread Graham Dumpleton
On May 10, 3:01 pm, mdipierro wrote: > Graham is right. The best we can do is catch > > try: >    data = src.read(size) > except: >    raise MySocketException > > and catch it later as I suggested above. What do you think? You can't technically do either of the things you have suggested, ie., c

Re: [web2py] Re: IOError, scalability concerned

2010-05-10 Thread Thadeus Burgess
My original concerns are eased, as I thought 120 people out of the 1000 that visited the site were getting errors [1], which seems to not be the case. If I were to go to all of the trouble to install nginx, I might as well stop using apache and install cherokee. I have been running my blog on cher

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread Graham Dumpleton
On May 10, 3:11 pm, Thadeus Burgess wrote: > Looking at the sites analytics, there are roughly about 1000 requests > a day, of which I will receive 100 of these tickets, give or take 20. If you want a quick solution, stick nginx in front as a proxy and you likely would eliminate the errors. Thi

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
strange. I get 50,000 requests/day (~10,000 non-static files) and not one error like this. I use apache_mod_wsgi. On May 10, 12:11 am, Thadeus Burgess wrote: > Looking at the sites analytics, there are roughly about 1000 requests > a day, of which I will receive 100 of these tickets, give or take

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
True. What about: def parse_get_post_vars(request, environ): try: request.body = copystream_progress(request) ### stores request body except: return False ... return True #in wsgibase ... if not parse_get_post_vars(request, environ): return [] On Ma

Re: [web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
Looking at the sites analytics, there are roughly about 1000 requests a day, of which I will receive 100 of these tickets, give or take 20. -- Thadeus On Mon, May 10, 2010 at 12:10 AM, Thadeus Burgess wrote: > What if there was an IO error with cgi.FieldStorage ? > > > > -- > Thadeus > > > >

Re: [web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
What if there was an IO error with cgi.FieldStorage ? -- Thadeus On Mon, May 10, 2010 at 12:06 AM, mdipierro wrote: > Yes. it is safe to ignore, but it still causes a minor slow down > because web2py issues a ticket and that may be unnecessary. > > On May 10, 12:01 am, Thadeus Burgess wro

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
Yes. it is safe to ignore, but it still causes a minor slow down because web2py issues a ticket and that may be unnecessary. On May 10, 12:01 am, Thadeus Burgess wrote: > So you are saying that a user with a slow connection (which consists > of a large portion of the sites user base) clicks reloa

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
How about replace? parse_get_post_vars(request, environ) with try: parse_get_post_vars(request, environ) except IOError: return [] On May 10, 12:01 am, mdipierro wrote: > Graham is right. The best we can do is catch > > try: >

Re: [web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
So you are saying that a user with a slow connection (which consists of a large portion of the sites user base) clicks reload, so their browser stops communicating with that request, so web2py catches it as an IOError... so then is this safe to ignore? Below is a log that I found in apache2.error.

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
Graham is right. The best we can do is catch try: data = src.read(size) except: raise MySocketException and catch it later as I suggested above. What do you think? Massimo On May 9, 11:52 pm, Graham Dumpleton wrote: > On May 10, 1:51 pm, Thadeus Burgess wrote: > > > Ticket. I usually se

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
gave this some more thought (still not enought) The problem is IOError which happens probably because users with slow connection click reload. Right now this is percaived by web2py an internal error, caught by main.py:441 and logged. Perhaps right before line 441 in main.py we should have somethi

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread Graham Dumpleton
On May 10, 1:51 pm, Thadeus Burgess wrote: > Ticket. I usually see them the next day when I check admin. > > No, it is usually just one IP but it happens to a lot of people at the > same time, scaled by the amount of traffic being put on the server. Then it is likely that the ISP or network all

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
How many requests/day do you have? I have never seen tickets like this. Clearly this is an issue. I run 2.5. Could this be a problem with 2.6 and web2py? Massimo On May 9, 10:51 pm, Thadeus Burgess wrote: > Ticket. I usually see them the next day when I check admin. > > No, it is usually just on

Re: [web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
Ticket. I usually see them the next day when I check admin. No, it is usually just one IP but it happens to a lot of people at the same time, scaled by the amount of traffic being put on the server. Roughly about 10% of the requests generate this error. > A user not waiting for a request to compl

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
Not sure about the problem but I had a few instances of people clicking reload a lot (and I mean a lot). So I use this: # drop incoming connections if IP make more than 10 connection attempts to port 80 within 100 seconds $IPT -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread mdipierro
Does this result in a ticket or console error? Do you get a lot of requests/sec from the same IP? On May 9, 9:49 pm, Graham Dumpleton wrote: > On May 10, 12:28 pm, Thadeus Burgess wrote: > > > What could possibly be causing this? > > A user not waiting for a request to complete before clicking o

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread Graham Dumpleton
On May 10, 12:28 pm, Thadeus Burgess wrote: > What could possibly be causing this? A user not waiting for a request to complete before clicking on another link or pressing reload. In other words, client dropped original connection. Graham > python 2.6 > web2py trunk > apache/mod_wsgi 2.6 > >

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
It is not a robot... I performed an IP lookup on some of the IP addresses that caused the error, and they come from ISP and typically in residential areas, and spread out. So this is a big issue since robots do not cause it. -- Thadeus On Sun, May 9, 2010 at 9:36 PM, Thadeus Burgess wrote: >

[web2py] Re: IOError, scalability concerned

2010-05-09 Thread Thadeus Burgess
Just some more information. Static files are served by apache (as to the book) Migrate = True copystream seems to allure to the fact a file is being uploaded (which is impossible because the application does not store any files) Might this be a robot / scanner attempting to upload things? If so,