On Thu, Jan 28, 2016 at 8:13 PM, Frank Millman <fr...@chagford.com> wrote: > Run the database handler in a separate thread. Use a queue.Queue to send > requests to the handler. Use an asyncio.Queue to send results back to the > caller, which can call 'await q.get()'. > > I ran a quick test and it seems to work. What do you think?
My gut feeling is that any queue can block at either get or put. Half of your operations are "correct", and the other half are "wrong". The caller can "await q.get()", which is the "correct" way to handle the blocking operation; the database thread can "jobqueue.get()" as a blocking operation, which is also fine. But your queue-putting operations are going to have to assume that they never block. Maybe you can have the database put its results onto the asyncio.Queue safely, but the requests going onto the queue.Queue could block waiting for the database. Specifically, this will happen if database queries come in faster than the database can handle them - quite literally, your jobs will be "blocked on the database". What should happen then? The other risk is that the wrong result will be queried (two async tasks put something onto the queue - which one gets the first result?), which could either be coped with by simple sequencing (maybe this happens automatically, although I'd prefer a mathematically-provable result to "it seems to work"), or by wrapping the whole thing up in a function/class. Both of these are risks seen purely by looking at the idea description, not at any sort of code. It's entirely possible I'm mistaken about them. But there's only one way to find out :) By the way, just out of interest... there's no way you can actually switch out the database communication for something purely socket-based, is there? PostgreSQL's protocol, for instance, is fairly straight-forward, and you don't *have* to use libpq; Pike's inbuilt pgsql module just opens a socket and says hello, and it looks like py-postgresql [1] is the same thing for Python. Taking something like that and making it asynchronous would be as straight-forward as converting any other socket-based code. Could be an alternative to all this weirdness. ChrisA [1] https://pypi.python.org/pypi/py-postgresql -- https://mail.python.org/mailman/listinfo/python-list
