Standard Asynchronous Python
After seeing David Mertz's talk at PyCon 2012, "Coroutines, event loops, and the history of Python generators" [1], I got thinking again about Python's expressive power for asynchronous programming. Generators, particularly with the addition of 'yield from' and 'return' in PEP 380 [2], allow us to write code that is executed "bit by bit" but still reads naturally. There are a number of frameworks that take advantage of this ability, but each is a little different -- enough so that there's negligible code re-use between these frameworks. I think that's a shame. I proposed a design PEP a while back [3] with the intent of defining a standard way of writing asynchronous code, with the goal of allowing code re-use and bringing users of the frameworks closer together. Ideally, we could have libraries to implement network protocols, database wrappers, subprocess execution, and so on, that would work in any of the available asynchronous frameworks. My proposal met with near-silence, and I didn't pursue it. Instead, I did what any self-respecting hacker would do - I wrote up a framework, uthreads [4], that implemented my idea. This was initially a simple trampoline scheduler, but I eventually refactored it to run atop Twisted, since that's what I use. To my knowledge, it's never been used. I'm considering re-drafting the PEP with the following changes: * De-emphasize the thread emulation aspects, and focus on code-portability issues: * callbacks vs. "blocking" calls (e.g., when accepting incoming connections on a socket, how is my code invoked?) * consistent access to primitives, regardless of framework (e.g., where's the function I call to branch execution?) * nested asynchronous methods * Account for PEP 380 (by making the StopIteration workarounds match PEP 380, and explicitly deprecating them after Python 3.3) * Look forward to a world with software transactional memory [5] by matching that API where appropriate As I get to work on the PEP, I'd like to hear any initial reactions to the idea. Dustin [1] https://us.pycon.org/2012/schedule/presentation/104/ [2] http://www.python.org/dev/peps/pep-0380 [3] http://code.google.com/p/uthreads/source/browse/trunk/microthreading-pep.txt [4] http://code.google.com/p/uthreads/ [5] https://bitbucket.org/pypy/pypy/raw/stm-thread/pypy/doc/stm.rst -- http://mail.python.org/mailman/listinfo/python-list
Re: Standard Asynchronous Python
The responses have certainly highlighted some errors in emphasis in my approach. * My idea is to propose a design PEP. (Steven, Dennis) I'm not at *all* suggesting including uthreads in the standard library. It's a toy implementation I used to develop my ideas. I think of this as a much smaller idea in the same vein as the DBAPI (PEP 249): a common set of expectations that allows portability. * I'd like to set aside the issue of threads vs. event-driven programming. There are legitimate reasons to do both, and the healthy ecosystem of frameworks for the latter indicates at least some people are interested. My idea is to introduce a tiny bit of coherence across those frameworks. * (Bryan) The Fibonacci example is a simple example of, among other things, a CPU-bound, recursive task -- something that many async frameworks don't handle fairly right now. I will add some text to call that out explicitly. * Regarding generators vs. coroutines (Bryan), I use the terms generator and generator function in the PEP carefully, as that's what the syntactic and runtime concepts are called in Python. I will include a paragraph distinguishing the two. I will need to take up the details of the idea with the developers of the async frameworks themselves, and get some agreement before actually proposing the PEP. However, among this group I'm interested to know whether this is an appropriate use of a design PEP. That's why I posted my old and flawed PEP text, rather than re-drafting first. Thanks for the responses so far! Dustin -- http://mail.python.org/mailman/listinfo/python-list
Re: Standard Asynchronous Python
Thanks for the second round of responses. I think this gives me some focus - concentrate on the API, talk to the framework developers, and start redrafting the PEP sooner rather than later. Thanks! Dustin -- http://mail.python.org/mailman/listinfo/python-list
Re: odt2sphinx 0.2.3 released
On Wed, Sep 12, 2012 at 10:06 AM, wrote: > ߒߤߒߡߜߦߡ ß ß§ And that's why you shouldn't let your kids play with your iPad :) Dustin -- http://mail.python.org/mailman/listinfo/python-list
Re: Dictionaries again - where do I make a mistake?
Lad wrote: > Sorting seems to be OK,. > the command > print key,val > prints the proper values > but I can not create Newdict to be sorted properly. > > Where do I make a mistake? > Thank you for help. Dictionaries are unordered -- the order in which items come out is unspecified. It's based on the details of their internal storage mechanism (a hash table), and you can't control it at all. If you need your pairs in a certain order, you'll have to use a list of tuples. Dustin -- http://mail.python.org/mailman/listinfo/python-list
httplib problems -- bug, or am I missing something?
I'm building an interface to Amazon's S3, using httplib. It uses a single object for multiple transactions. What's happening is this: HTTP > PUT /unitest-temp-1161039691 HTTP/1.1 HTTP > Date: Mon, 16 Oct 2006 23:01:32 GMT HTTP > Authorization: AWS <>:KiTWRuq/6aay0bI2J5DkE2TAWD0= HTTP > (end headers) HTTP < HTTP/1.1 200 OK HTTP < content-length: 0 HTTP < x-amz-id-2: 40uQn0OCpTiFcX+LqjMuzG6NnufdUk/.. HTTP < server: AmazonS3 HTTP < x-amz-request-id: FF504E8FD1B86F8C HTTP < location: /unitest-temp-1161039691 HTTP < date: Mon, 16 Oct 2006 23:01:33 GMT HTTPConnection.__state before response.read: Idle HTTPConnection.__response: closed? False length: 0 reading response HTTPConnection.__state after response.read: Idle HTTPConnection.__response: closed? False length: 0 ..later in the same connection.. HTTPConnection.__state before putrequest: Idle HTTPConnection.__response: closed? False length: 0 HTTP > DELETE /unitest-temp-1161039691 HTTP/1.1 HTTP > Date: Mon, 16 Oct 2006 23:01:33 GMT HTTP > Authorization: AWS <>:a5OizuLNwwV7eBUhha0B6rEJ+CQ= HTTP > (end headers) HTTPConnection.__state before getresponse: Request-sent HTTPConnection.__response: closed? False length: 0 File "/usr/lib64/python2.4/httplib.py", line 856, in getresponse raise ResponseNotReady() If the first request does not precede it, the second request is fine. To avoid excessive memory use, I'm calling request.read(16384) repeatedly, instead of just calling request.read(). This seems to be key to the problem -- if I omit the 'amt' argument to read(), then the last line of the first request reads HTTPConnection.__response: closed? True length: 0 and the later call to getresponse() doesn't raise ResponseNotReady. Looking at the source for httplib.HTTPResponse.read, self.close() gets called in the latter (working) case, but not in the former (non-working). It would seem sensible to add 'if self.length == 0: self.close()' to the end of that function (and, in fact, this change makes the whole thing work), but this comment makes me hesitant: # we do not use _safe_read() here because this may be a .will_close # connection, and the user is reading more bytes than will be provided # (for example, reading in 1k chunks) What's going on here? Is this a bug I should report, or am I missing something about how one should use httplib? Thanks for any assistance. Dustin -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling functions
Tommy Grav wrote: > I have a small program that goes something like this > > def funcA() : pass > def funcB() : pass > def funcC() : pass > > def determine(f): > t = f() > return t > > What I would like to do is be able to > > n = determine(funcA) > m = determine(funcB) > > But I can't really figure out how to do this (I think it is > possible :) Except for the spaces after the def's at the top (are those legal?), it should work as written. determine(funcA) results in 'f' being bound to 'funcA'; then 't = f()' results in 'funcA' being called, and its resulting being bound to 't'; 'determine' returns that result, and it's bound to 'n'. Is that not what you wanted? Dustin -- http://mail.python.org/mailman/listinfo/python-list
Redux: Allowing 'return obj' in generators
This question was first brought up in October of 2005[1], and was included in
the "Unresolved Issues" section of my microthreading PEP, which I have quietly
withdrawn from consideration due to lack of community interest.
PEP 255 says
Q. Then why not allow an expression on "return" too?
A. Perhaps we will someday. In Icon, "return expr" means both "I'm
done", and "but I have one final useful value to return too, and
this is it". At the start, and in the absence of compelling uses
for "return expr", it's simply cleaner to use "yield" exclusively
for delivering values.
As those of you who looked at my PEP or are familiar with some of the
implementations will realize, microthreaded functions are syntactically
generator functions, but semantically act as regular functions. There
is a well-defined meaning to 'return x' in such a function: take the
value of x, and use it in the expression where this function was called.
For example:
def read_integer(sock):
txt = yield sock.readline().strip()
try:
return int(txt)
except:
raise AppProtocolError("Expected an integer")
The implementation of the syntax would be similar to that of an
expressionless 'return', but supplying the expression_list to the
StopIteration's 'args' -- this is described quite well in Piet Delport's
post[2].
Given this use-case (and note that I chose an example that will exercise
the interactions of try/except blocks with the StopIteration behavior),
is it time to revisit this issue? BDFL said:
I urge you to leave well enough alone. There's room for extensions
after people have built real systems with the raw material provided by
PEP 342 and 343.[3]
and Nick Coghlan said (to applause from GvR):
I'm starting to think we want to let PEP 342 bake for at least one
release cycle before deciding what (if any) additional behaviour
should be added to generators.[4]
I think we have a decent number of implementations in the wild now
(I have learned of Christopher Stawarz's 'multitask'[5] since last
posting my PEP). With 2.5.1 out, might I suggest this is worth
reconsidering for the 2.6 release?
Dustin
[1]
http://www.python.org/dev/summary/2005-10-01_2005-10-15/#allowing-return-obj-in-generators
[2] http://mail.python.org/pipermail/python-dev/2005-October/056957.html
[3] http://mail.python.org/pipermail/python-dev/2005-October/057119.html
[4] http://mail.python.org/pipermail/python-dev/2005-October/057133.html
[5] http://o2s.csail.mit.edu/o2s-wiki/multitask
--
http://mail.python.org/mailman/listinfo/python-list
