On Saturday, September 1, 2012 3:28:52 PM UTC-4, Laszlo Nagy wrote: > > Hi > > > > > > does running on tornado imply that you would not consider twisted > > > http://twistedmatrix.com ? > > > > > > If not, twisted has exactly this capability hiding long running > > > queries on whatever db's behind deferToThread(). > > All right, I was reading its documentation > > > > http://twistedmatrix.com/documents/10.1.0/api/twisted.internet.threads.deferToThread.html > > > > It doesn't tell too much about it: "Run a function in a thread and > > return the result as a Deferred.". >
You can find more documentation here: http://twistedmatrix.com/documents/current/core/howto/threading.html Also, Twisted has dedicated APIs for interacting with databases asynchronously: http://twistedmatrix.com/documents/current/core/howto/rdbms.html Additionally, there is a non-blocking (rather than thread-based) implementation of the above API available for PostgreSQL: http://pypi.python.org/pypi/txpostgres > > > Run a function but in what thread? Does it create a new thread for every > > invocation? In that case, I don't want to use this. My example case: 10% > > from 100 requests/second deal with a database. But it does not mean that > > one db-related request will do a single db API call only. They will > > almost always do more: start transaction, parse and open query, fetch > > with cursor, close query, open another query etc. then commit > > transaction. 8 API calls to do a quick fetch + update (usually under > > 100msec, but it might be blocked by another transaction for a while...) > > So we are talking about 80 database API calls per seconds at least. It > > would be insane to initialize a new thread for each invocation. And > > wrapping these API calls into a single closure function is not useful > > either, because that function would not be able to safely access the > > state that is stored in the main thread. Unless you protet it with > > locks. But it is whole point of async I/O server: to avoid using slow > > locks, expensive threads and context switching. > > > > Maybe, deferToThread uses a thread pool? But it doesn't say much about > > it. (Am I reading the wrong documentation?) BTW I could try a version > > that uses a thread pool. > > > > It is sad, by the way. We have async I/O servers for Python that can be > > used for large number of clients, but most external modules/extensions > > do not support their I/O loops. Including the extension modules of the > > most popular databases. So yes, you can use Twisted or torandoweb until > > you do not want to call *some* API functions that are blocking. (By > > *some* I mean: much less blocking than non-blocking, but quite a few.) > > We also have synchronous Python servers, but we cannot get rid of the > > GIL, Python threads are expensive and slow, so they cannot be used for a > > large number of clients. And finally, we have messaging services/IPC > > like zeromq. They are probably the most expensive, but they scale very > > well. But you need more money to operate the underlying hardware. I'm > > starting to think that I did not get a quick answer because my use case > > (100 clients) fall into to the "heavy weight" category, and the solution > > is to invest more in the hardware. :-) > > > > Thanks, > > > > Laszlo -- http://mail.python.org/mailman/listinfo/python-list
