Re: How to write a non blocking SimpleHTTPRequestHandler ?
Thank you Amirouch. I was hoping to use something very simple and already provided by the standard library. If I can fix the script, the better. If the script can't be fixed, then I'll switch to another library (I already have one in mind). -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
Hello Irmen, > Why? If you have multiple threads serving some requests at the same time, > doesn't that > already achieve your goal? Having multiple requests running at a time is one thing. Making them non-blocking is another. That's how I understand it. > In other words, have you tried what you describe above? Yes I already tried that, it's in the source code I posted if you look closely you'll see that I am inherting from the mixin. The server still hangs sometime, until I send it the SIGINT signal. It then sort of drops the hanging socket and resumes normal operations. > (make sure you close the connection correctly or you'll be hogging a thread > which may > eventually make the server non responsive) > > Irmen I didn't see any such code in the tests that came with the standard python library (which I took as a starting point example to write my script). Maybe it's already taken care of by some lower-level internal stuff ? -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
On Tuesday, February 3, 2015 at 10:27:11 AM UTC+1, Marko Rauhamaa wrote: > The standard library and nonblocking can't be used in the same sentence. Thanks Marko. It's a lost cause then. I am thinking about switching to one of the following : - CherryPy - Bottle - circuits - Quixote - Weblayer. If anybody has a pointer to an already made comparison I'm a taker. Thanks for your help ! -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
> > Thanks Marko. It's a lost cause then. > > You trimmed out the part where he mentioned asyncio. :) > > ChrisA IIRC asyncio is python 3 only and I'm not ready yet to make the leap. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
Thanks Chris, it's only a matter of time, I'll eventually make the transition to python3 when I'll learn it well enough. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
On Tuesday, February 3, 2015 at 12:35:32 PM UTC+1, Marko Rauhamaa wrote: > So far I've been happy with select.epoll(), socket.socket() and ten > fingers. > Marko There's already software written to take care of much of the HTTP stuff protocol stuff, the headers etc. I wouldn't rewrite it. I prefer to monkey patch parts of existing code rather then rewrite all of it. But your comment is interesting because, as I understand it, a non-blocking web server is simply a matter of setting timeouts on sockets, catch the exceptions and move on. I don't know why wouldn't that be possible with python stdlib ? > The standard library and nonblocking can't be used in the same sentence. ? -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
On Tuesday, February 3, 2015 at 3:17:37 PM UTC+1, Amirouche Boubekki wrote: > What you want is to prevent the socket to wait indefinetly for data (based on > strace output) which is done with socket.setblocking/settimeout [1] Exactly ! thanks for taking time to reading my original post a second time. Maybe I didn't express my problem well enough and the discussion has maybe drifted to an XYProblem type discussion :) I was about to say that most webframeworks for python already take care of the problem I'm facing, so I certainly confused people with improper formulation. > There is contentbrowser [2] which is somekind of web proxy. It's using werkzeug and it seems to be a full web application. I just want something that is right above sockets and selects. All that my script is doing is grabbing some json data from an external url provided by the browser and return a formatted page showing the json data in human-readable HTML pages. Nothing fancy. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
On Tuesday, February 3, 2015 at 3:17:37 PM UTC+1, Amirouche Boubekki wrote:
> What you want is to prevent the socket to wait indefinetly for data (based on
> strace output) which is done with socket.setblocking/settimeout [1].
> asynchronous (asyncio) is something else, and you would still need to handle
> blocking I think.
I have installed Faulthandler, a beautiful tool written by Victor "Haypo"
Stinner, and thanks to it I could determine precisely where the program hangs.
It is in ssl.py::SSLSocket::read
def read(self, len=1024):
"""Read up to LEN bytes and return them.
Return zero-length string on EOF."""
try:
>return self._sslobj.read(len) >*< python hangs on this line
except SSLError, x:
if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
return ''
else:
raise
>From the traceback given by faulthandler it seems to me that the problem isn't
>from my webserver trying to receive connections from clients, but from my
>server (acting as a client) trying a request on a distant server (https URL ->
>use of ssl.py). Here's the traceback :
1 Current thread 0x7fb9cb41f700 (most recent call first):
2 File "/usr/lib/python2.7/ssl.py", line 160 in read
3 File "/usr/lib/python2.7/ssl.py", line 241 in recv
4 File "/usr/lib/python2.7/socket.py", line 447 in readline
5 File "/usr/lib/python2.7/httplib.py", line 365 in _read_status
6 File "/usr/lib/python2.7/httplib.py", line 407 in begin
7 File "/usr/lib/python2.7/httplib.py", line 1034 in getresponse
8 File
"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py",
line 353 in _make_request
9 File
"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py",
line 518 in urlopen
10 File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py",
line 370 in send
11 File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py",
line 573 in send
12 File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py",
line 461 in request
13 File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py",
line 473 in get
14 File "/usr/local/lib/python2.7/dist-packages/infomaniak/infomaniak.py",
line 29 in getFlux
15 File "/usr/local/lib/python2.7/dist-packages/infomaniak/server.py",
line 52 in do_GET
16 File "/usr/lib/python2.7/BaseHTTPServer.py", line 328 in
handle_one_request
17 File "/usr/lib/python2.7/BaseHTTPServer.py", line 340 in handle
18 File "/usr/lib/python2.7/SocketServer.py", line 649 in __init__
19 File "/usr/lib/python2.7/SocketServer.py", line 334 in finish_request
20 File "/usr/lib/python2.7/SocketServer.py", line 321 in process_request
21 File "/usr/lib/python2.7/SocketServer.py", line 295 in
_handle_request_noblock
22 File "/usr/lib/python2.7/SocketServer.py", line 238 in serve_forever
23 File "/usr/local/lib/python2.7/dist-packages/infomaniak/server.py",
line 71 in
24 File "/usr/bin/infomaniak", line 2 in
25
If you look at line 13, it shows that the server is actually doing a get to an
external URL via the requests library, which is itself relying on urllib3,
which in turn is using httplib.py
Below is the code of the last three functions to have been called, in
chronoligical order :
In socket.py::_fileobject::readline
[...]
while True:
try:
data = self._sock.recv(self._rbufsize) #<
except error, e:
if e.args[0] == EINTR:
continue
raise
if not data:
break
nl = data.find('\n')
if nl >= 0:
nl += 1
buf.write(data[:nl])
self._rbuf.write(data[nl:])
del data
break
buf.write(data)
return buf.getvalue()
[...]
In ssl.py::SSLSocket::recv
def recv(self, buflen=1024, flags=0):
if self._sslobj:
if flags != 0:
raise ValueError(
"non-zero flags not allowed in calls to recv() on %s" %
self.__class__)
return self.read(buflen) #<---
else:
return self._sock.recv(buflen, flags)
In ssl.py::SSLSocket::read
def read(self, len=1024):
"""Read up to LEN bytes and return them.
bottle.py app doesn't timeout properly ?
Hello,
I have written a script using bottle.py. The app works fine most of times.
Sometimes though, the server takes time to respond and the web browser
eventually drops the connection to the server after a certain time (timeout),
showing this page :
"""
Connection reset
The connection to the server was reset while the page was loading.
This site may be temporarily unavailable or too busy. Try again in a few
moments.
If you can not load any pages, check your computer's network connection.
If your computer or network is protected by a firewall or proxy, make sure
that Firefox is permitted to access the web.
"""
In the app, I have setup faulthandler
(https://pypi.python.org/pypi/faulthandler/2.4) in my application to respond to
SIGUSER1. When I trigger the signal (while the server is hung), here's the
traceback I get. It shows precisely where is my application hanging :
Current thread 0x7fb1a0bfe700 (most recent call first):
File "/usr/lib/python2.7/socket.py", line 447 in readline
File "/usr/lib/python2.7/wsgiref/simple_server.py", line 116 in handle
File "/usr/lib/python2.7/SocketServer.py", line 649 in __init__
File "/usr/lib/python2.7/SocketServer.py", line 334 in finish_request
File "/usr/lib/python2.7/SocketServer.py", line 321 in process_request
File "/usr/lib/python2.7/SocketServer.py", line 295 in _handle_request_noblock
File "/usr/lib/python2.7/SocketServer.py", line 238 in serve_forever
File "/usr/local/lib/python2.7/dist-packages/infomaniak/bottle.py", line 2680
in run
File "/usr/local/lib/python2.7/dist-packages/infomaniak/bottle.py", line 3048
in run
File "/usr/local/lib/python2.7/dist-packages/infomaniak/server.py", line 69
in
File "/usr/bin/infomaniak", line 2 in
Instead of the readline call to timeout after a certain amount of time, it
seems it gets stuck there forever. When I strace my app it, I find it stuck at
the recvfrom system call :
root@audio-mon[10.10.10.82] ~ # service infomaniak status
infomaniak is running with pid 2149
root@audio-mon[10.10.10.82] ~ # strace -fp 2149
Process 2149 attached - interrupt to quit
recvfrom(7, ^C
Process 2149 detached
root@audio-mon[10.10.10.82] ~ #
It is stuck there for hours (7 hours now).
How can I fix this ? this behaviour is also hard to reproduce, I don't know
what triggers it. The app was working fine for almost a month, no problem at
all.
Here's code :
#!/usr/bin/env python
#-*- encoding=utf-8 -*-
# infomaniak.getFlux() et infomaniak.login()
import infomaniak
# Pour la localisation des fichiers HTML
import pkg_resources
import logging
import ConfigParser
# pour attrapper l'exception requests.exceptions.ConnectionError
import requests
# pour faire tourner le schmilblick
import bottle
# service infomaniak reload
import signal
# service infomaniak trace
import faulthandler
cp = ConfigParser.ConfigParser()
cp.read("/etc/infomaniak.conf")
PORT = cp.getint("SERVER","PORT")
USER = cp.get("LOGIN","USER")
PASS = cp.get("LOGIN","PASS")
LOG_LEVEL = cp.get("LOG","LOG_LEVEL")
logging.basicConfig(filename='/var/log/infomaniak/main.log',
level=getattr(logging,LOG_LEVEL),
format='%(asctime)s [%(levelname)s] %(message)s'
)
logging.info("= START ==")
logging.info("Reading config file Ok.. Serving on port %s" % (PORT))
def reload_conf(signal,frame):
logging.info("= RELOAD ==")
logging.info("Reading config file Ok.. Serving on port %s" % (PORT))
cp = ConfigParser.ConfigParser()
cp.read("/etc/infomaniak.conf")
LOG_LEVEL = cp.get("LOG","LOG_LEVEL")
logging.getLogger().setLevel(getattr(logging,LOG_LEVEL))
logging.info("LOG LEVEL %s" % (LOG_LEVEL))
signal.signal(signal.SIGUSR2,reload_conf)
faulthandler.register(signal.SIGUSR1,file=open("/var/log/infomaniak/main.log","a"))
@bottle.route("/")
def get(url):
# print "thing",url
#
logging.info("--")
# logging.info("[%s:%s] get %s" %
(self.client_address[0],self.client_address[1],self.path))
# MUST BE OF THE FORM :
# https://statslive.infomaniak.com/radio/config/formatsflux.php/g3377s3i4402
return infomaniak.getFlux(url.lstrip("/"),USER,PASS)
logged_in = False
attempts = 1
while (logged_in == False) :
logging.info("login...(tentative %d)" % attempts)
try:
infomaniak.login(USER,PASS)
logged_in = True
except requests.exceptions.ConnectionError,e:
logging.critical("Impossible de se connecter au serveur infomaniak")
logging.critical(e)
attempts+=1
logging.info("login OK (tentatives : %d)" % attempts)
bottle.run(host='',port=PORT)
logging.info("= STOP ==")
Any ideas on how to debug this or try to reproduce it ? anyone encountered
similar behaviou
Re: bottle.py app doesn't timeout properly ?
Here's a traceback I generated by catching a SIGINT and printing an exception : Traceback (most recent call last): File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock self.process_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request self.finish_request(request, client_address) File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__ self.handle() File "/usr/lib/python2.7/wsgiref/simple_server.py", line 116, in handle self.raw_requestline = self.rfile.readline() File "/usr/lib/python2.7/socket.py", line 447, in readline data = self._sock.recv(self._rbufsize) KeyboardInterrupt Last line shows where the app was when the SIGINT happened, same socket.py's readline function. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
Hello, I wanted to make a little update on this thread. The problem is solved, and while debugging my application I learned that it is actually possible to have multithreaded or multi-process web application in python using only the standard library. A longer explanation, along with minimal working code and some UML diagrams explaining the insides of the standard python library modules involved in basic web applications can be found here : http://ychaouche.wikispot.org/HowBottleAppsWork -- https://mail.python.org/mailman/listinfo/python-list
Re: How to write a non blocking SimpleHTTPRequestHandler ?
Here is a simple multi-threaded python web application that uses only the
stanard library modules :
#!/usr/bin/env python
#-*- encoding=utf-8 -*-
import SimpleHTTPServer
import BaseHTTPServer
import SocketServer
class MyServer(SocketServer.ThreadingMixIn,BaseHTTPServer.HTTPServer):
pass
class Handler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write("Hello world \n")
IP,PORT = "",8010
server = MyServer((IP,PORT),Handler)
server.serve_forever()
It will send Hello world followed by a newline (so that when you invoke curl on
the terminal it will nicely put the shell prompt a newline). You can simulate a
non-responsive client with this little script :
import socket
import requests
# This will leave an open a connection to our app.
conn = socket.create_connection(('localhost',8010))
# This will never get anything until conn.close() is called.
print requests.get('http://localhost:8010').text.strip()
If you don't inherit from the ThreadingMixIn, the application will get stuck
when you launch the client script and any further requests (use curl or wget
for example) will simply be postponed until client.py is killed.
If you inherit from ThreadingMixIn, the application will nicely run the request
handler on a new thread, making way for further requests to be handled.
--
https://mail.python.org/mailman/listinfo/python-list
