Re: Async client for PostgreSQL?
On 8/31/12 7:17 PM, Laszlo Nagy wrote: Is there any extension for Python that can do async I/O for PostgreSQL with tornadoweb's ioloop? Something like: class MainHandler(tornado.web.RequestHandler): @tornado.web.asynchronous def get(self): pg_connection.(long_taking_query_sql,params,callback=self.on_query_opened) def on_query_opened(self, query): self.write(process_rows(query)) self.finish() What would be an alternative? The theoretical problem: suppose there are 100 clients (web browsers) connected to the server with keep alive connections. They are doing long-polls and they are also sending/receiving events (with short response times). Each web browser has an associated state stored on the server side, in the memory (as an object tree). The state is bound to the client with a session id. Most requests will have to be responded with small amounts of data, calculated from the session state, or queried from the database. Most database queries are simple, running for about 100msec. But a few of them will run for 1sec or more. Number of requests ending in database queries is relatively low (10/sec). Other requests can be responded must faster. but they are much more frequent (100/sec, that is. 1 request/sec/client). There is a big global cache full of (Python) objects. Their purpose is to reduce the number of database queries. These objects in the global cache are emitting events to other objects found in the client sessions. Generally, it is not possible to tell what request will end in a database query. Multi-threading is not an option because number of clients is too high (running 100 threads is not good). This is why I decided to use anyc I/O. Tornadoweb looks good for most requirements: async i/o, store session state in objects etc. The biggest problem is that psycopg is not compatible with this model. If I use blocking I/O calls inside a request handler, then they will block all other requests most of the time, resulting in slow response times. What would be a good solution for this? Thanks, Laszlo Hi does running on tornado imply that you would not consider twisted http://twistedmatrix.com ? If not, twisted has exactly this capability hiding long running queries on whatever db's behind deferToThread(). Brute force I would pack the db access into a twisted run web-service, forking work out in twisted either with deferToThread() or if you want to take advantage of using processes instead of threads thus being able to tap all cores, then have a look at ampoule at https://launchpad.net/ampoule - be aware though that ampoule has a 64k limit on what can be passed around. Werner -- http://mail.python.org/mailman/listinfo/python-list
Re: Async client for PostgreSQL?
On 9/1/12 9:28 AM, Laszlo Nagy wrote: Hi does running on tornado imply that you would not consider twisted http://twistedmatrix.com ? If not, twisted has exactly this capability hiding long running queries on whatever db's behind deferToThread(). All right, I was reading its documentation http://twistedmatrix.com/documents/10.1.0/api/twisted.internet.threads.deferToThread.html It doesn't tell too much about it: "Run a function in a thread and return the result as a Deferred.". Run a function but in what thread? Does it create a new thread for every invocation? In that case, I don't want to use this. My example case: 10% from 100 requests/second deal with a database. But it does not mean that one db-related request will do a single db API call only. They will almost always do more: start transaction, parse and open query, fetch with cursor, close query, open another query etc. then commit transaction. 8 API calls to do a quick fetch + update (usually under 100msec, but it might be blocked by another transaction for a while...) So we are talking about 80 database API calls per seconds at least. It would be insane to initialize a new thread for each invocation. And wrapping these API calls into a single closure function is not useful either, because that function would not be able to safely access the state that is stored in the main thread. Unless you protet it with locks. But it is whole point of async I/O server: to avoid using slow locks, expensive threads and context switching. Maybe, deferToThread uses a thread pool? But it doesn't say much about it. (Am I reading the wrong documentation?) BTW I could try a version that uses a thread pool. It is sad, by the way. We have async I/O servers for Python that can be used for large number of clients, but most external modules/extensions do not support their I/O loops. Including the extension modules of the most popular databases. So yes, you can use Twisted or torandoweb until you do not want to call *some* API functions that are blocking. (By *some* I mean: much less blocking than non-blocking, but quite a few.) We also have synchronous Python servers, but we cannot get rid of the GIL, Python threads are expensive and slow, so they cannot be used for a large number of clients. And finally, we have messaging services/IPC like zeromq. They are probably the most expensive, but they scale very well. But you need more money to operate the underlying hardware. I'm starting to think that I did not get a quick answer because my use case (100 clients) fall into to the "heavy weight" category, and the solution is to invest more in the hardware. :-) Thanks, Laszlo Laszlo: Hmm, I was suggesting that you could replace the whole DB driver with a webservice implemented with twisted, if you rule out threads then with ampoule doing it with a process pool and consume this webservice with the tornado side asynchronously. production level example thread pool based DB API: Just to give you some ballpark figures, I'm running a game server with a daily peak of about 1500 parallel permanent connections and 50k games played every day (avg game duration 13min, peak request frequency close to 100req/sec) with a lot of statistics going into a MySQL DB on US$2k worth of hardware. Twisted as basis sitting atop FreeBSD, started the latest version in March, its running since then, no restarts, no reboots, no problems. production level example process pool based PDF production: Or for another implementation I'm running a webservice based PDF production (probably as blocking as services can come) for a Java based business app with twisted/ampoule, this is as stable as the game server. HTH, Werner -- http://mail.python.org/mailman/listinfo/python-list
Re: Import Problem on WIndows py3
On 9/4/12 9:49 AM, jimmyli1...@gmail.com wrote: I have a main program and a 3rd party module. Trying to import colorama, where colorama is a folder with files in it, returns an ImportError: No module named colorama. How should I import folders? Do you have a (empty) __init__.py file present in this particular directory? Werner -- http://mail.python.org/mailman/listinfo/python-list
Re: parallel programming in Python
For such tasks my choice would be twisted combined with ampoule. Let's you spread out work to whatever amount of processes you desire, maxing out whatever iron you're sitting on.. HTH, Werner http://twistedmatrix.com/trac/ https://launchpad.net/ampoule On 29.05.2012 16:43, Jabba Laci wrote: Hehe, I just asked this question a few days ago but I didn't become much cleverer: http://www.gossamer-threads.com/lists/python/python/985701 Best, Laszlo On Thu, May 10, 2012 at 2:14 PM, Jabba Laci wrote: Hi, I would like to do some parallel programming with Python but I don't know how to start. There are several ways to go but I don't know what the differences are between them: threads, multiprocessing, gevent, etc. I want to use a single machine with several cores. I want to solve problems like this: iterate over a loop (with millions of steps) and do some work at each step. The steps are independent, so here I would like to process several steps in parallel. I want to store the results in a global list (which should be "synchronised"). Typical use case: crawl webpages, extract images and collect the images in a list. What's the best way? Thanks, Laszlo <>-- http://mail.python.org/mailman/listinfo/python-list
Re: Does anyone know of a python tapi module
On 7/1/11 11:15 AM, Alister Ware wrote: The subject probably say is all but to elaborate. I am looking for a way to communicate with a tapi driver for a PBX so I can experiment with creating some CTI (Computer Telephony Integration) software. I used TAPI since its inception for quite a few projects, but am not aware of anything pythonic. Attempting to cough up an interface layer I would resort to using Chris Sells tfx wrapper fro TAPI, which helps a lot keeping things in check and becoming overly complex. HTH, Werner http://books.google.com/books?id=3M_mIvtdGqUC&pg=PA82&lpg=PA82&dq=chris+sell+tfx&source=bl&ots=5UhAxbFTym&sig=7hRn0oUbyZsm_yyDvASKD-AT1jo&hl=en&ei=w2UOTsPlBJOisAP57JyrDg&sa=X&oi=book_result&ct=result&resnum=1&ved=0CBkQ6AEwAA -- http://mail.python.org/mailman/listinfo/python-list
Re: I am not able to open IDLE in python on Vista.
Ever tried to run it as Administrator (right click, Run as Administrator...) Werner Am 01.03.2011 17:12, schrieb Jayneil Dalal: This is the error message I get when I try to run Pyhon on Vista: unable to create user config directory C:\Users\Jayneil\.idlerc Check path and permissions. Exiting! So please help me out!. Thank you. <>-- http://mail.python.org/mailman/listinfo/python-list
Re: Python CPU
You probably heard of the infamous FORTH chips like the Harris RTX2000, or ShhBoom, which implemented a stack oriented very low power design before there were FPGAs in silicon. To my knowledge the RTX2000 is still used for space hardened application and if I search long enough I might fine the one I had sitting in my cellar. The chip was at that time so insanely fast that it could produce video signals with FORTH programs driving the IO pins. Chuck Moore, father of FORTH developed the chip on silicon in FORTH itself. Due to the fact, that the instruction sets of a FORTH machine, being a very general stack based von Neumann system, I believe that starting with an RTX2000 (which should be available in VHDL) one could quite fast be at a point where things make sense, meaning not going for the 'fastest' ever CPU but for the advantage of having a decent CPU programmable in Python sitting on a chip with a lot of hardware available. Another thing worth to mention in this context is for sure the work available on http://www.myhdl.org/doku.php. Werner On 4/3/11 3:46 AM, Dan Stromberg wrote: On Sat, Apr 2, 2011 at 5:10 PM, Gregory Ewing mailto:greg.ew...@canterbury.ac.nz>> wrote: Brad wrote: I've heard of Java CPUs. Has anyone implemented a Python CPU in VHDL or Verilog? Not that I know of. I've had thoughts about designing one, just for the exercise. It's doubtful whether such a thing would ever be of practical use. Without as much money as Intel has to throw at CPU development, it's likely that a Python chip would always be slower and more expensive than an off-the-shelf CPU running a tightly-coded interpreter. It could be fun to speculate on what a Python CPU might look like, though. One with the time and inclination could probably do a Python VM in an FPGA, no? Though last I heard, FPGA's weren't expected to increase in performance as fast as general-purpose CPU's. -- http://mail.python.org/mailman/listinfo/python-list
Re: "Daemonizing" an application.
Hi Might be an overkill, but have a look at twisted, http://www.twistedmatrix.com I usually use the spread package for structured communication between partners via localhost What are the best practices to do this ? Examples in a well known Pyhon app I could hack ? Is it possible with standard packages only ? Have fun, Werner -- http://mail.python.org/mailman/listinfo/python-list
Re: Twisted or Tornado?
Hi On 3/1/13 1:39 AM, Michael Torrie wrote: On 02/28/2013 05:28 PM, Jake Angulo wrote: My requirements for this framework in descending order: 1) Easy to use API 2) Widely available documentation / Examples / Community contributions 3) Feature-wise - kinda most that you commonly need is there I'm happy with twisted programming browser based multi-player games for several years now. I wouldn't say that the learning curve was steep, rather the opposite, but the gain from a clean async programming model as it is implemented in twisted really makes the difference for me, not only in writing games, but even more in maintaining them long term. It's also interesting to see, that this clean async model of twisted and its host of already provided protocols allowed me to implement solutions in notoriously problematic areas like overall stability, fault tolerance, load induced behavior, server analysis, multicore/multimachine capabilities and hot deployment with ease. Cheers, Werner -- http://mail.python.org/mailman/listinfo/python-list
Re: To Thread or not to Thread....?
Hi I see quite a few alleys to go down when stuck with such types of problems, but instead of listing and discussing them have a look at a quite complete discussion and comparison of the various async programming options available at http://syncless.googlecode.com Also have a look at the presentation at http://syncless.googlecode.com/svn/trunk/doc/slides_2010-11-29/pts_coro_2010-11-29.html If I were in your shoes I would solve the problem in stackless, if exchanging the python interpreter is not possible then Twisted would be my second choice, having done a lot of work with it (see the NMEA classes for serial ports). I don't know if syncless handles other types of fd's like serial ports, I've never played with it. The monkey patching of syncless might pose other problems in your case. HTH, Werner Am 01.12.2010 14:48, schrieb James Mills: Surely I2C is just a serial-like interface and one should be able to do async I/O on it ? The use of threads is not necessary here and the GIL doesn't become a problem in async I/O anyway. I only use threads for operating that might block (not for I/O). cheers James On Wed, Dec 1, 2010 at 7:24 PM, Antoine Pitrou wrote: On Wed, 1 Dec 2010 02:45:50 + Jack Keegan wrote: Hi there, I'm currently writing an application to control and take measurements during an experiments. This is to be done on an embedded computer running XPe so I am happy to have python available, although I am pretty new to it. The application basically runs as a state machine, which transitions through it's states based on inputs read in from a set of general purpose input/output (GPIO) lines. So when a certain line is pulled low/high, do something and move to another state. All good so far and since I get through main loop pretty quickly, I can just do a read of the GPIO lines on each pass through the loop and respond accordingly. However, in one of the states I have to start reading in, and storing frames from a camera. In another, I have to start reading accelerometer data from an I2C bus (which operates at 400kHz). I haven't implemented either yet but I would imagine that, in the case of the camera data, reading a frame would take a large amount of time as compared to other operations. Therefore, if I just tried to read one (or one set of) frames on each pass through the loop then I would hold up the rest of the application. Conversely, as the I2C bus will need to be read at such a high rate, I may not be able to get the required data rate I need even without the camera data. This naturally leads me to think I need to use threads. As I am no expert in either I2C, cameras, python or threading I thought I would chance asking for some advice on the subject. Do you think I need threads here or would I be better off using some other method. I was previously toying with the idea of using generators to create weightless threads (as detailed in http://www.ibm.com/developerworks/library/l-pythrd.html) for reading the GPIOs. Do you think this would work in this situation? The main question IMO: the I2C bus operates at 400kHz, but how much received data can it buffer? That will give you a hint as to how much latency you can tolerate. I don't think soft threads would work at all, since they wouldn't allow overlapping between frame reading and other ops. If frame reading releases the GIL (as any properly implemented IO primitive in Python should), then at least you can try using OS threads. Then, depending on the tolerable latency for I2C operation, you can try to run it as an OS thread, or a separate process (if running as a separate process, make sure it cannot block while sending IO to the master process). Regards Antoine. -- http://mail.python.org/mailman/listinfo/python-list <>-- http://mail.python.org/mailman/listinfo/python-list
Re: python ide for ubuntu
Eclipse with pydev (great debugging) does the trick nicely, free of charge and throws in some other goodies (JScript/HTML/XML editing) too. I use the EE for Java developer version http://www.eclipse.org/downloads/packages/eclipse-ide-java-ee-developers/heliosr Install pydev from Menu Help/Software Updates After ten years of Python coding and suffering thru most of the commercial products like VisualPython, Komodo, Wings, asf I heartily recommend Eclipse nowadays. If you want to stay as pythonesque as possible you could go with SPE which uses wxPython and integrates with Blender, although this project seems to be stalled. Werner On 12.08.2010 04:15, Bhanu Kumar wrote: Hi All, Is there any good free python IDE available in Ubuntu? thanks, -Bhanu <>-- http://mail.python.org/mailman/listinfo/python-list
Re: Deferring a function call
For me nothing beats using twisted ( http://www.twistedmatrix.com ) with its clean implementation of deferreds and a host of other useful things for simulations besides being the 'Swiss Army Knife' of networking in Python. If you throw in stackless ( http://www.stackless.com ) simulations with extremely high counts of participants and/or fine granularity distributed across multiple machines are fun to write, operate and maintain. Werner On 10/18/10 6:21 PM, TomF wrote: I'm writing a simple simulator, and I want to schedule an action to occur at a later time. Basically, at some later point I want to call a function f(a, b, c). But the values of a, b and c are determined at the current time. One way way to do this is to keep a list of entries of the form [[TIME, FN, ARGS]...] and at simulated time TIME do: apply(FN, ARGS) Aside from the fact that apply is deprecated, it seems like there should be a cleaner (possibly more Pythonic) way to do this. Ideas? -Tom -- http://mail.python.org/mailman/listinfo/python-list
