[Python-Dev] PEP 492: async/await in Python; version 4

[tds...@gmail.com]

Hi,

still watching progress here. Read all posts and changes.

Everything improved and I know it is a lot of work. Thx for doing this.

But I still think this PEP goes to far.

1. To have "native coroutines" and await, __await__ is very good and
useful. Also for beginners to see and mark coroutines are a different
concept than generators. Also this is easy to learn. Easier as to
explain why a generator is in this case a coroutine and so on.

2. I still don't like to sprinkle async everywhere. At all we don't need
it for the first step.
We can handle coroutines similar to generators, when there is a await in
it then it is one. Same as for yield. Or to be more explicit, if it is
marked as one with @coroutine it is enough. But then it makes also sense
to do the same for generators with @generator. We should be consistent here.

3. async with is not needed, there are rare use cases for it and every
can be written with a try: except: finally:
Every async framework lived without it over years. No problem because
for the seldom need try: ... was enough.

4. async for can be implemented with a while loop. For me this is even
more explicit and clear. Every time I see the async for I try to find
out what is done in async manner. For what I do an implicit await ?
Also in most of my uses cases it was enough to produce Futures in a
normal loop and yield (await) for them.
Even for database interfaces. Most of the time they do prefetching under
the hood and I don't have to care at all.

5. For async with/for a lot of new special methods are needed all only
prefixed with "a". Easy to confuse with "a" for abstract. Often used
to prefix abstract classes. Still think __async_iter__, __async_enter__
is better for this. Yes more to write but not so easy to confuse with
the sync __iter__ or __enter__, ... And matches more to I must use async
to call them.


I am not new to the async land, have done a lot of stuff with twisted
and even looked at tornado.
Also has tried to teach some people the asnc world. This is not easy
and you learn most only by doing and using it.

For my final conclusion, we should not rush all this into the language.
Do it step by step and also help other Python implementations to support
it. For now only a very low percentage are at Python 3.4.
And if you compare the statistics in PyPi, you see most are still at
Python 2.7 and using async frameworks like twisted/tornado.
We do such a fundamental language change only because a few people need
it for a niche in the whole Python universe?
We confuse the rest with new stuff they never need?

Even the discussion on python-dev suggests there is some time needed
to finalize all this.

We forget to address the major problems here. How can someone in a
"sync" script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.

And for all this we still can't tell them "oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5". It does not and it is only most useful for networking stuff
nothing more.

Don't get me wrong, I like async programming and I think it is very
useful. But had to learn not everyone thinks so and most only want
to solve there problems in an easy way and not get a new one called
"async".


Now I shut up. Go to my normal mode and be quiet and read. ;-)



Regards,

Wolfgang





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492: async/await in Python; version 4

[Wolfgang]

Hi,

still watching progress here. Read all posts and changes.

Everything improved and I know it is a lot of work. Thx for doing this.

But I still think this PEP goes to far.

1. To have "native coroutines" and await, __await__ is very good and
useful. Also for beginners to see and mark coroutines are a different
concept than generators. Also this is easy to learn. Easier as to
explain why a generator is in this case a coroutine and so on.

2. I still don't like to sprinkle async everywhere. At all we don't need
it for the first step.
We can handle coroutines similar to generators, when there is a await in
it then it is one. Same as for yield. Or to be more explicit, if it is
marked as one with @coroutine it is enough. But then it makes also sense
to do the same for generators with @generator. We should be consistent here.

3. async with is not needed, there are rare use cases for it and every
can be written with a try: except: finally:
Every async framework lived without it over years. No problem because
for the seldom need try: ... was enough.

4. async for can be implemented with a while loop. For me this is even
more explicit and clear. Every time I see the async for I try to find
out what is done in async manner. For what I do an implicit await ?
Also in most of my uses cases it was enough to produce Futures in a
normal loop and yield (await) for them.
Even for database interfaces. Most of the time they do prefetching under
the hood and I don't have to care at all.

5. For async with/for a lot of new special methods are needed all only
prefixed with "a". Easy to confuse with "a" for abstract. Often used
to prefix abstract classes. Still think __async_iter__, __async_enter__
is better for this. Yes more to write but not so easy to confuse with
the sync __iter__ or __enter__, ... And matches more to I must use async
to call them.


I am not new to the async land, have done a lot of stuff with twisted
and even looked at tornado.
Also has tried to teach some people the asnc world. This is not easy
and you learn most only by doing and using it.

For my final conclusion, we should not rush all this into the language.
Do it step by step and also help other Python implementations to support
it. For now only a very low percentage are at Python 3.4.
And if you compare the statistics in PyPi, you see most are still at
Python 2.7 and using async frameworks like twisted/tornado.
We do such a fundamental language change only because a few people need
it for a niche in the whole Python universe?
We confuse the rest with new stuff they never need?

Even the discussion on python-dev suggests there is some time needed
to finalize all this.

We forget to address the major problems here. How can someone in a
"sync" script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.

And for all this we still can't tell them "oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5". It does not and it is only most useful for networking stuff
nothing more.

Don't get me wrong, I like async programming and I think it is very
useful. But had to learn not everyone thinks so and most only want
to solve there problems in an easy way and not get a new one called
"async".


Now I shut up. Go to my normal mode and be quiet and read. ;-)



Regards,

Wolfgang





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Koos Zevenhoven]

tds333  gmail.com  gmail.com> writes:

> 
> Hi,
> 
> still watching progress here. Read all posts and changes.
> 
> Everything improved and I know it is a lot of work. Thx for doing this.
> 
> But I still think this PEP goes to far.
> 
> [...]
> 
> We forget to address the major problems here. How can someone in a
> "sync" script use this async stuff easy. How can async and sync stuff
> cooperate and we don't need to rewrite the world for async stuff.
> How can a normal user access the power of async stuff without rewriting
> all his code. So he can use a simple asyc request library in his code.
> How can a normal user learn and use all this in an easy way.
> 
> [...]

Hi Wolfgang,

You may want to see what I just posted on python-ideas. What I wrote about 
is related to several things you mention, and might provide a remedy.

-- Koos



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Jim J. Jewett]

On Fri May 1 23:58:26 CEST 2015, Yury Selivanov wrote:


>>> Yes, you can't use 'yield from' in __exit__/__enter__
>>> in current Python.

>> What do you mean by "can't use"?

> It probably executed without errors, but it didn't run the
> generators.

True.  But it did return the one created by __enter__, so it
could be bound to a variable and iterated within the block.

There isn't an easy way to run the generator created by __exit__,
and I'm not coming up with any obvious scenarios where it would
be a sensible thing to do (other than using "with" on a context
manager that *does* return a future instead of finishing).

That said, I'm still not seeing why the distinction is so important
that we have to enforce it at a language level, as opposed to letting
the framework do its own enforcement.  (And if the reason is
performance, then make the checks something that can be turned off,
or offer a fully instrumented loop as an alternative for debugging.)

 Is the intent to do anything more than preface execution with:
 import asynchio.coroutines
 asynchio.coroutines._DEBUG = True

>>> If you enable it after you import asyncio,
>>> then asyncio itself won't be instrumented.

>> Why does asynchio itself have to wrapped?  Is that really something
>> normal developers need to debug, or is it only for developing the
>> stdlib itself?  

> Yes, normal developers need asyncio to be instrumented,
> otherwise you won't know what you did wrong when you
> called some asyncio code without 'await' for example.

I'll trust you that it *does* work that way, but this sure sounds to
me as though the framework isn't ready to be frozen with syntax, and
maybe not even ready for non-provisional stdlib inclusion.

I understand that the disconnected nature of asynchronous tasks makes
them harder to debug.  I heartily agree that the event loop should
offer some sort of debug facility to track this.

But the event loop is supposed to be pluggable.  Saying that this
requires not merely a replacement, or even a replacement before events
are added, but a replacement made before python ever even loads the
default version ...

That seems to be much stronger than sys.settrace -- more like
instrumenting the ceval loop itself.  And that is something that
ordinary developers shouldn't have to do.


-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Oscar Benjamin]

On 30 April 2015 at 09:50, Arnaud Delobelle  wrote:
>>
>> I'm flexible about how we name 'async def' functions.  I like
>> to call them "coroutines", because that's what they are, and
>> that's how asyncio calls them.  It's also convenient to use
>> 'coroutine-object' to explain what is the result of calling
>> a coroutine.
>
> I'd like the object created by an 'async def' statement to be called a
> 'coroutine function' and the result of calling it to be called a
> 'coroutine'.

That would be an improvement over the confusing terminology in the PEP
atm. The PEP proposes to name the inspect functions
inspect.iscoroutine() and inspect.iscoroutinefunction(). According to
the PEP iscoroutine() identifies "coroutine objects" and
iscoroutinefunction() identifies "coroutine functions" -- a term which
is not defined in the PEP but presumably means what the PEP calls a
"coroutine" in the glossary.

Calling the async def function an "async function" and the object it
returns a "coroutine" makes for the clearest terminology IMO (provided
the word coroutine is not also used for anything else). It would help
to prevent both experienced and new users from confusing the two
related but necessarily distinct concepts. Clearly distinct
terminology makes it easier to explain/discuss something if nothing
else because it saves repeating definitions all the time.


--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492: async/await in Python; version 5

[Yury Selivanov]

Hi python-dev,


Updated version of the PEP is below.

Quick summary of changes:

1. set_coroutine_wrapper and get_coroutine_wrapper functions
are now thread-specific (like settrace etc).

2. Updated Abstract & Rationale sections.

3. RuntimeWarning is always raised when a coroutine wasn't
awaited on. This is in addition to what 'set_coroutine_wrapper'
will/can do.

4. asyncio.async is renamed to asyncio.ensure_future; it will
be deprecated in 3.5.

5. Uses of async/await in CPython codebase are documented.

6. Other small edits and updates.


Thanks,
Yury



PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015,
  05-May-2015


Abstract


The growth of Internet and general connectivity has triggered the
proportionate need for responsive and scalable code.  This proposal
aims to answer that need by making writing explicitly asynchronous,
concurrent Python code easier and more Pythonic.

It is proposed to make *coroutines* a proper standalone concept in
Python, and introduce new supporting syntax.  The ultimate goal
is to help establish a common, easily approachable, mental
model of asynchronous programming in Python and make it as close to
synchronous programming as possible.

We believe that the changes proposed here will help keep Python
relevant and competitive in a quickly growing area of asynchronous
programming, as many other languages have adopted, or are planning to
adopt, similar features: [2]_, [5]_, [6]_, [7]_, [8]_, [10]_.


Rationale and Goals
===

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:

* It is easy to confuse coroutines with regular generators, since they
  share the same syntax; this is especially true for new developers.

* Whether or not a function is a coroutine is determined by a presence
  of ``yield``  or ``yield from`` statements in its *body*, which can
  lead to unobvious errors when such statements appear in or disappear
  from function body during refactoring.

* Support for asynchronous calls is limited to expressions where
  ``yield`` is allowed syntactically, limiting the usefulness of
  syntactic features, such as ``with`` and ``for`` statements.

This proposal makes coroutines a native Python language feature, and
clearly separates them from generators.  This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library.  This also enables
linters and IDEs to improve static code analysis and refactoring.

Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.


Specification
=

This proposal introduces new syntax and semantics to enhance coroutine
support in Python.

This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380).  Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).

From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax.  *coroutine* is used in contexts where both
definitions are applicable.


New Coroutine Declaration Syntax


The following new syntax is used to declare a *native coroutine*::

async def read_data(db):
pass

Key properties of *coroutines*:

* ``async def`` functions are always coroutines, even if they do not
  contain ``await`` expressions.

* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
  expressions in an ``async`` function.

* Internally, two new code object flags were introduced:

  - ``CO_COROUTINE`` is used to enable runtime detection of
*coroutines* (and migrating existing code).

  - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
(defined with new syntax.)

  All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
  ``CO_GENERATOR`` flags set.

* Regular generators, when called, return a *generator object*;
  similarly, coroutines return a *coroutine object*.

* ``StopIteration`` exceptions are not propagated out of coroutines,
  and are replaced with a ``RuntimeError``.  

Re: [Python-Dev] PEP 492: What is the real goal?

[Yury Selivanov]

Hi Oscar,

I've updated the PEP with some fixes of the terminology:
https://hg.python.org/peps/rev/f156b272f860

I still think that 'coroutine functions' and 'coroutines'
is a better pair than 'async functions' and 'coroutines'.

First, it's similar to existing terminology for generators.
Second, it's less confusing.  With pep492 at some point,
using generators to implement coroutines won't be a wide
spread practice, so 'async def' functions will be the only
language construct that returns them.

Yury

On 2015-05-05 12:01 PM, Oscar Benjamin wrote:

On 30 April 2015 at 09:50, Arnaud Delobelle  wrote:

I'm flexible about how we name 'async def' functions.  I like
to call them "coroutines", because that's what they are, and
that's how asyncio calls them.  It's also convenient to use
'coroutine-object' to explain what is the result of calling
a coroutine.

I'd like the object created by an 'async def' statement to be called a
'coroutine function' and the result of calling it to be called a
'coroutine'.

That would be an improvement over the confusing terminology in the PEP
atm. The PEP proposes to name the inspect functions
inspect.iscoroutine() and inspect.iscoroutinefunction(). According to
the PEP iscoroutine() identifies "coroutine objects" and
iscoroutinefunction() identifies "coroutine functions" -- a term which
is not defined in the PEP but presumably means what the PEP calls a
"coroutine" in the glossary.

Calling the async def function an "async function" and the object it
returns a "coroutine" makes for the clearest terminology IMO (provided
the word coroutine is not also used for anything else). It would help
to prevent both experienced and new users from confusing the two
related but necessarily distinct concepts. Clearly distinct
terminology makes it easier to explain/discuss something if nothing
else because it saves repeating definitions all the time.


--
Oscar


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

Hi Wolfgang,


On 2015-05-05 7:27 AM, Wolfgang wrote:

Hi,

[..]

Even the discussion on python-dev suggests there is some time needed
to finalize all this.


I'd say that:

80% of the recent discussion of the PEP is about terminology.
10% is about whether we should have __future__ import or not.



We forget to address the major problems here. How can someone in a
"sync" script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.


asyncio and twisted answered these questions ;) The answer is
that you have to write async implementations.

gevent has a different answer, but greenlents/stackless is
something that will never be merged in CPython and other
implementations.



And for all this we still can't tell them "oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5". It does not and it is only most useful for networking stuff
nothing more.


"networking stuff", and in particular, web, is a huge
part of current Python usage.  Please don't underestimate
that.


Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ABCs - Re: PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Mon, May 4, 2015 at 10:35 AM, Jim J. Jewett  wrote:

> Which reminds me ... *should* the "await" keyword work with any future,
> or is it really intentionally restricted to use with a single library
> module and 3rd party replacements?
>

You can make any Future type work with await by adding an __await__ method.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Paul Moore]

On 5 May 2015 at 19:25, Yury Selivanov  wrote:
> On 2015-05-05 7:27 AM, Wolfgang wrote:
>> Even the discussion on python-dev suggests there is some time needed
>> to finalize all this.
>
> I'd say that:
>
> 80% of the recent discussion of the PEP is about terminology.
> 10% is about whether we should have __future__ import or not.

But the terminology discussion appears to revolve around people
finding the various concepts involved in asyncio (particularly the new
PEP, but also to an extent the existing implementation) confusing. I
can confirm, having tried to work through the asyncio docs, that the
underlying concepts and how they are explained, are confusing to an
outsider.

That's not to say that everything needs to be beginner-friendly, but
it *does* mean that it's hard for the wider Python community to
meaningfully comment, or evaluate or sanity-check the design. We're
left with a sense of "trust us, it makes sense if you need it,
everyone else can ignore it".

Personally, I feel as if PEP 492 is looking a little premature - maybe
the focus should be on making asyncio more accessible first, and
*then* adding syntax. You can argue that the syntax is needed to help
make async more accessible - but if that's the case then the
terminology debates and confusion are clear evidence that it's not
succeeding in that goal. Of course, that's based on my perception of
one of the goals of the PEP as being "make coroutines and asyncio more
accessible", If the actual goals are different, my conclusion is
invalid.

>> We forget to address the major problems here. How can someone in a
>> "sync" script use this async stuff easy. How can async and sync stuff
>> cooperate and we don't need to rewrite the world for async stuff.
>> How can a normal user access the power of async stuff without rewriting
>> all his code. So he can use a simple asyc request library in his code.
>> How can a normal user learn and use all this in an easy way.
>
> asyncio and twisted answered these questions ;) The answer is
> that you have to write async implementations.

Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?

Many people use requests for their web access. There are good reasons
for this. Are you saying that until someone steps up and writes an
async implementation of requests, I have to make a choice - requests
or asyncio? Unfortunately, I can't see myself choosing asyncio in that
situation. Which again means that asyncio becomes "something that the
average user can't use". Which in turn further entrenches it as a
specialist-only tool.

As another example, in Twisted I could use defer_to_thread to
integrate Oracle database access into a twisted application (that's
what the twisted database stuff did under the hood). Can I do that
with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
to that?

>> And for all this we still can't tell them "oh the async stuff solves
>> the multiprocessing problem of Python learn it and switch to version
>> 3.5". It does not and it is only most useful for networking stuff
>> nothing more.
>
> "networking stuff", and in particular, web, is a huge
> part of current Python usage.  Please don't underestimate
> that.

Without async versions of requests and similar, how much of a chunk of
the networking/web area will asyncio take? (Genuine question, I have
no idea). And how much extra will this PEP add? Those may not be fair
questions (and even if they are fair, the answers are probably
unknowable), but as an outsider, I feel only the costs of the asyncio
implementation (a new library that I don't understand, and now a
relatively large amount of new syntax and special methods I have to
ignore because they don't make sense to me). That's OK, but I think I
am being reasonable to ask for some sense of the level of benefits
others are getting to balance out the costs I incur.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Guido van Rossum]

On Tue, May 5, 2015 at 9:48 AM, Yury Selivanov 
wrote:

> Hi Oscar,
>
> I've updated the PEP with some fixes of the terminology:
> https://hg.python.org/peps/rev/f156b272f860
>
> I still think that 'coroutine functions' and 'coroutines'
> is a better pair than 'async functions' and 'coroutines'.
>

Yes. This subtopic is now closed for further debate (please).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Jim J. Jewett]

On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492.

Where are the following over-simplifications wrong?

(1)  The PEP is intended for use (almost exclusively) with
asychronous IO and a scheduler such as the asynchio event loop.

(2)  The new syntax is intended to make it easier to recognize when
a task's execution may be interrupted by arbitrary other tasks, and
the interrupted task therefore has to revalidate assumptions about
shared data.

With threads, CPython can always suspend a task between op-codes,
but with a sufficiently comprehensive loop (and sufficiently
coooperative tasks), tasks *should* only be suspended when they
make an explicit request to *wait* for an answer, and these points
*should* be marked syntactically.

(3)  The new constructs explicitly do NOT support any sort of
concurrent execution within a task; they are for use precisely
when otherwise parallel subtasks are being linearized by pausing
and waiting for the results.


Over-simplifications 4-6 assume a world with standardized futures
based on concurrent.futures, where .result either returns the
result or raises the exception (or raises another exception about
timeout or cancellation).

[Note that the actual PEP uses iteration over the results of a new
__await__ magic method, rather than .result on the object itself.
I couldn't tell whether this was for explicit marking, or just for
efficiency in avoiding future creation.]

(4)  "await EXPR" is just syntactic sugar for EXPR.result

except that, by being syntax, it better marks locations where
unrelated tasks might have a chance to change shared data.

[And that, as currently planned, the result of an await isn't
actually the result; it is an iterator of results.]

(5)  "async def" is just syntactic sugar for "def", 

except that, by being syntax, it better marks the signatures of
functions and methods where unrelated tasks might have a chance
to change shared data after execution has already begun.

(5A) As the PEP currently stands, it is also a promise that the
function will NOT produce a generator used as an iterator; if a
generator-iterator needs to wait for something else at some point,
that will need to be done differently.

I derive this limitation from
   "It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function."

but I don't understand how this limitation works with things like a
per-line file iterator that might need to wait for the file to
be initially opened.

(6)  async with EXPR as VAR:

would be equivalent to:

with EXPR as VAR:

except that
  __enter__() would be replaced by next(await __enter__()) # __enter__().result
  __exit__() would be replaced by  next(await __exit__())  # __exit__().result


(7)  async for elem in iter:

would be shorthand for:

for elem in iter:
elem = next(await elem) # elem.result



-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Brett Cannon]

On Tue, May 5, 2015 at 3:14 PM Paul Moore  wrote:

> On 5 May 2015 at 19:25, Yury Selivanov  wrote:
> > On 2015-05-05 7:27 AM, Wolfgang wrote:
> >> Even the discussion on python-dev suggests there is some time needed
> >> to finalize all this.
> >
> > I'd say that:
> >
> > 80% of the recent discussion of the PEP is about terminology.
> > 10% is about whether we should have __future__ import or not.
>
> But the terminology discussion appears to revolve around people
> finding the various concepts involved in asyncio (particularly the new
> PEP, but also to an extent the existing implementation) confusing. I
> can confirm, having tried to work through the asyncio docs, that the
> underlying concepts and how they are explained, are confusing to an
> outsider.
>
> That's not to say that everything needs to be beginner-friendly, but
> it *does* mean that it's hard for the wider Python community to
> meaningfully comment, or evaluate or sanity-check the design. We're
> left with a sense of "trust us, it makes sense if you need it,
> everyone else can ignore it".
>

Watch David Beazley's talk from PyCon this year and you can watch him
basically re-implement asyncio on stage in under 45 minutes. It's not as
complicated as it seems when you realize there is an event loop driving
everything (which people have been leaving out of the conversation since it
doesn't tie into the syntax directly).


>
> Personally, I feel as if PEP 492 is looking a little premature - maybe
> the focus should be on making asyncio more accessible first, and
> *then* adding syntax.


I think this ties the concept of adding syntax to Python to make
coroutine-based programming easier too much to asyncio; the latter is just
an implementation of the former. This PEP doesn't require asyncio beyond
the fact that will be what provides the initial event loop in the stdlib.


> You can argue that the syntax is needed to help
> make async more accessible - but if that's the case then the
> terminology debates and confusion are clear evidence that it's not
> succeeding in that goal.


Perhaps, but arguing about the nitty-gritty details of something doesn't
automatically lead to a clearer understanding of the higher level concept.
Discussing how turning a steering wheel in a car might help you grasp how
cars turn, but it isn't a requirement to get "turn the wheel left to make
the car go left".


> Of course, that's based on my perception of
> one of the goals of the PEP as being "make coroutines and asyncio more
> accessible", If the actual goals are different, my conclusion is
> invalid.
>

I think the goal is "make coroutines easier to use" and does not directly
relate to asyncio.


>
> >> We forget to address the major problems here. How can someone in a
> >> "sync" script use this async stuff easy. How can async and sync stuff
> >> cooperate and we don't need to rewrite the world for async stuff.
> >> How can a normal user access the power of async stuff without rewriting
> >> all his code. So he can use a simple asyc request library in his code.
> >> How can a normal user learn and use all this in an easy way.
> >
> > asyncio and twisted answered these questions ;) The answer is
> > that you have to write async implementations.
>
> Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
> but that seems to be callback-based rather than coroutine-based?
>

Yep.


>
> Many people use requests for their web access. There are good reasons
> for this. Are you saying that until someone steps up and writes an
> async implementation of requests, I have to make a choice - requests
> or asyncio?


I believe so; you need something to implement __await__. This is true in
any language that implements co-routines.

Unfortunately, I can't see myself choosing asyncio in that
> situation. Which again means that asyncio becomes "something that the
> average user can't use". Which in turn further entrenches it as a
> specialist-only tool.
>

You forgot to append "... yet" to that statement. Just because something
isn't available out of the box without some effort to support doesn't mean
it will never happen, else there would be absolutely no Python 3 users out
there.


>
> As another example, in Twisted I could use defer_to_thread to
> integrate Oracle database access into a twisted application (that's
> what the twisted database stuff did under the hood). Can I do that
> with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
> to that?
>
> >> And for all this we still can't tell them "oh the async stuff solves
> >> the multiprocessing problem of Python learn it and switch to version
> >> 3.5". It does not and it is only most useful for networking stuff
> >> nothing more.
> >
> > "networking stuff", and in particular, web, is a huge
> > part of current Python usage.  Please don't underestimate
> > that.
>
> Without async versions of requests and similar, how much of a chunk of
> the networking/web area will asyncio take? (Genuine quest

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

Paul,

On 2015-05-05 3:14 PM, Paul Moore wrote:

On 5 May 2015 at 19:25, Yury Selivanov  wrote:

On 2015-05-05 7:27 AM, Wolfgang wrote:

Even the discussion on python-dev suggests there is some time needed
to finalize all this.

I'd say that:

80% of the recent discussion of the PEP is about terminology.
10% is about whether we should have __future__ import or not.

But the terminology discussion appears to revolve around people
finding the various concepts involved in asyncio (particularly the new
PEP, but also to an extent the existing implementation) confusing. I
can confirm, having tried to work through the asyncio docs, that the
underlying concepts and how they are explained, are confusing to an
outsider.


I agree. We have to improve asyncio docs in this area.



That's not to say that everything needs to be beginner-friendly, but
it *does* mean that it's hard for the wider Python community to
meaningfully comment, or evaluate or sanity-check the design. We're
left with a sense of "trust us, it makes sense if you need it,
everyone else can ignore it".

Personally, I feel as if PEP 492 is looking a little premature - maybe
the focus should be on making asyncio more accessible first, and
*then* adding syntax. You can argue that the syntax is needed to help
make async more accessible - but if that's the case then the
terminology debates and confusion are clear evidence that it's not
succeeding in that goal. Of course, that's based on my perception of
one of the goals of the PEP as being "make coroutines and asyncio more
accessible", If the actual goals are different, my conclusion is
invalid.


Again, PEP 492 is not only for asyncio. *Any* framework can
use it, including Twisted.

As for terminology, I view this discussion differently.  It's
not about the technical details (Python has asymmetric
coroutines, that's it), but rather on how to disambiguate
coroutines implemented with generators and yield-from, from
new 'async def' coroutines.  I can't see any fundamental
problem with the PEP behind such discussions.




We forget to address the major problems here. How can someone in a
"sync" script use this async stuff easy. How can async and sync stuff
cooperate and we don't need to rewrite the world for async stuff.
How can a normal user access the power of async stuff without rewriting
all his code. So he can use a simple asyc request library in his code.
How can a normal user learn and use all this in an easy way.

asyncio and twisted answered these questions ;) The answer is
that you have to write async implementations.

Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?

Many people use requests for their web access. There are good reasons
for this. Are you saying that until someone steps up and writes an
async implementation of requests, I have to make a choice - requests
or asyncio? Unfortunately, I can't see myself choosing asyncio in that
situation. Which again means that asyncio becomes "something that the
average user can't use". Which in turn further entrenches it as a
specialist-only tool.


There is aiohttp library [1], which provides a client API
similar to requests.

And if you want to write high performance networking server
in python3 you *will* choose asyncio (or gevent/twisted
in python2).

And PEP 492 is aimed to make this whole async stuff more
accessible to an average user.



As another example, in Twisted I could use defer_to_thread to
integrate Oracle database access into a twisted application (that's
what the twisted database stuff did under the hood). Can I do that
with asyncio? Will the syntax in the PEP help, hinder or be irrelevant
to that?


You can use 'loop.run_in_executor' in asyncio. It returns a
future that you can await on.  You can also provide a nice
facade for your Oracle-database code that provides a nice
API but uses asyncio thread executor behind the scenes.




And for all this we still can't tell them "oh the async stuff solves
the multiprocessing problem of Python learn it and switch to version
3.5". It does not and it is only most useful for networking stuff
nothing more.

"networking stuff", and in particular, web, is a huge
part of current Python usage.  Please don't underestimate
that.

Without async versions of requests and similar, how much of a chunk of
the networking/web area will asyncio take? (Genuine question, I have
no idea).


There are some things (like websockets) that are hard
to implement correctly in existing frameworks like django
and flask.  And these kind of things are becoming more
and more important.  Languages like Go were designed
specifically to allow writing efficient


And how much extra will this PEP add? Those may not be fair
questions (and even if they are fair, the answers are probably
unknowable), but as an outsider, I feel only the costs of the asyncio
implementation (a new library that I don't understand, and now a
relatively large amount o

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Yury Selivanov]

On 2015-05-05 3:40 PM, Jim J. Jewett wrote:

On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492.

Where are the following over-simplifications wrong?

(1)  The PEP is intended for use (almost exclusively) with
asychronous IO and a scheduler such as the asynchio event loop.


Yes. You can also use it for UI loops.  Basically, anything
that can call your code asynchronously.



(2)  The new syntax is intended to make it easier to recognize when
a task's execution may be interrupted by arbitrary other tasks, and
the interrupted task therefore has to revalidate assumptions about
shared data.

With threads, CPython can always suspend a task between op-codes,
but with a sufficiently comprehensive loop (and sufficiently
coooperative tasks), tasks *should* only be suspended when they
make an explicit request to *wait* for an answer, and these points
*should* be marked syntactically.

(3)  The new constructs explicitly do NOT support any sort of
concurrent execution within a task; they are for use precisely
when otherwise parallel subtasks are being linearized by pausing
and waiting for the results.


Yes.



Over-simplifications 4-6 assume a world with standardized futures
based on concurrent.futures, where .result either returns the
result or raises the exception (or raises another exception about
timeout or cancellation).

[Note that the actual PEP uses iteration over the results of a new
__await__ magic method, rather than .result on the object itself.
I couldn't tell whether this was for explicit marking, or just for
efficiency in avoiding future creation.]

(4)  "await EXPR" is just syntactic sugar for EXPR.result

except that, by being syntax, it better marks locations where
unrelated tasks might have a chance to change shared data.

[And that, as currently planned, the result of an await isn't
actually the result; it is an iterator of results.]


I'm not sure how to comment on (4).  Perhaps I don't
understand some notation that you're using.  If anything,
it's more of a syntactic sugar for 'yield from EXPR'.



(5)  "async def" is just syntactic sugar for "def",

except that, by being syntax, it better marks the signatures of
functions and methods where unrelated tasks might have a chance
to change shared data after execution has already begun.


It also sets "CO_COROUTINE | CO_GENERATOR" flags, that
are very important.



(5A) As the PEP currently stands, it is also a promise that the
function will NOT produce a generator used as an iterator; if a
generator-iterator needs to wait for something else at some point,
that will need to be done differently.

I derive this limitation from
"It is a ``SyntaxError`` to have ``yield`` or ``yield from``
 expressions in an ``async`` function."

but I don't understand how this limitation works with things like a
per-line file iterator that might need to wait for the file to
be initially opened.


Per-line file iterator can be implemented with __aiter__,
__anext__ protocol.  __aiter__ is a coroutine, you can
open/start reading your file there.



(6)  async with EXPR as VAR:

would be equivalent to:

 with EXPR as VAR:

except that
   __enter__() would be replaced by next(await __enter__()) # __enter__().result
   __exit__() would be replaced by  next(await __exit__())  # __exit__().result


I'm not sure I understand what you mean by
"next(await EXPR)" notation.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Nathaniel Smith]

On May 5, 2015 12:40 PM, "Jim J. Jewett"  wrote:
>
>
> On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492.
>
> Where are the following over-simplifications wrong?
>
[...snip...]
>
> [Note that the actual PEP uses iteration over the results of a new
> __await__ magic method, rather than .result on the object itself.
> I couldn't tell whether this was for explicit marking, or just for
> efficiency in avoiding future creation.]
>
> (4)  "await EXPR" is just syntactic sugar for EXPR.result
>
> except that, by being syntax, it better marks locations where
> unrelated tasks might have a chance to change shared data.
>
> [And that, as currently planned, the result of an await isn't
> actually the result; it is an iterator of results.]

This is where you're missing a key idea. (And I agree that more high-level
docs are very much needed!) Remember that this is just regular single
threaded python code, so just writing EXPR.result cannot possibly cause the
current task to pause and another one to start running, and then magically
switch back somehow when the result does become available. Imagine trying
to implement a .result attribute that does that -- it's impossible.

Writing 'x = await socket1.read(1)' is actually equivalent to writing a
little loop like:

while True:
# figure out what we need to happen to make progress
needed = "data from socket 1"
# suspend this function,
# and send the main loop a message telling it what we need
reply = (yield needed)
# okay, the main loop woke us up again
# let's see if they've sent us back what we asked for
if reply.type == "data from socket 1":
# got it!
x = reply.payload
break
else:
# if at first you don't succeed...
continue

(Now stare at the formal definition of 'yield from' until you see how it
maps onto the above... And if you're wondering why we need a loop, think
about the case where instead of calling socket.read we're calling http.get
or something that requires multiple steps to complete.)

So there actually is semantically no iterator here -- the thing that looks
like an iterator is actually the chatter back and forth between the
lower-level code and the main loop that is orchestrating everything. Then
when that's done, it returns the single result.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Paul Moore]

(Yury gave similar responses, so (a) I'll just respond here, and (b)
it's encouraging that you both responded so quickly with the same
message)

On 5 May 2015 at 20:44, Brett Cannon  wrote:
>> That's not to say that everything needs to be beginner-friendly, but
>> it *does* mean that it's hard for the wider Python community to
>> meaningfully comment, or evaluate or sanity-check the design. We're
>> left with a sense of "trust us, it makes sense if you need it,
>> everyone else can ignore it".
>
> Watch David Beazley's talk from PyCon this year and you can watch him
> basically re-implement asyncio on stage in under 45 minutes. It's not as
> complicated as it seems when you realize there is an event loop driving
> everything (which people have been leaving out of the conversation since it
> doesn't tie into the syntax directly).

I'll watch that - it should be fun. But I have seen things like that
before, and I've got an idea how to write an event loop. You're right
that it's easy to lose track of the fundamentally simple idea in all
the complex discussions. To me that feels like a peculiar failure of
the abstraction, in that in some circumstances it makes things feel
*more* complex than they are :-)

>> Personally, I feel as if PEP 492 is looking a little premature - maybe
>> the focus should be on making asyncio more accessible first, and
>> *then* adding syntax.
>
> I think this ties the concept of adding syntax to Python to make
> coroutine-based programming easier too much to asyncio; the latter is just
> an implementation of the former. This PEP doesn't require asyncio beyond the
> fact that will be what provides the initial event loop in the stdlib.

It's very hard to separate coroutines from asyncio, because there's no
other example (not even a toy one) to reason about.

It would probably be helpful to have a concrete example of a basic
event loop that did *nothing* but schedule tasks. No IO waiting or
similar, just scheduling. I have a gut feeling that event loops are
more than just asyncio, but without examples to point to it's hard to
keep a focus on that fact. And even harder to isolate "what is an
event loop mechanism" from "what is asyncio specific". For example,
asyncio.BaseEventLoop has a create_connection method. That's
*obviously* not a fundamental aspect of a generic event loop, But
call_soon (presumably) is. Having a documented "basic event loop"
interface would probably help emphasise the idea than event loops
don't have to be asyncio. (Actually, what *is* the minimal event loop
interface that is needed for the various task/future mechanisms to
work, independently of asyncio? And what features of an event loop etc
are needed for the PEP, if it's being used outside of asyncio?)

I guess the other canonical event loop use case is GUI system message
dispatchers.

>> You can argue that the syntax is needed to help
>> make async more accessible - but if that's the case then the
>> terminology debates and confusion are clear evidence that it's not
>> succeeding in that goal.
>
> Perhaps, but arguing about the nitty-gritty details of something doesn't
> automatically lead to a clearer understanding of the higher level concept.
> Discussing how turning a steering wheel in a car might help you grasp how
> cars turn, but it isn't a requirement to get "turn the wheel left to make
> the car go left".

Fair point. If only I could avoid driving into walls :-)

>> Of course, that's based on my perception of
>> one of the goals of the PEP as being "make coroutines and asyncio more
>> accessible", If the actual goals are different, my conclusion is
>> invalid.
>
> I think the goal is "make coroutines easier to use" and does not directly
> relate to asyncio.

OK. But in that case, some examples using a non-asyncio toy "just
schedule tasks" event loop might help.

>> Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
>> but that seems to be callback-based rather than coroutine-based?
>
> Yep.

... and so you can't use it with async/await?

>> Many people use requests for their web access. There are good reasons
>> for this. Are you saying that until someone steps up and writes an
>> async implementation of requests, I have to make a choice - requests
>> or asyncio?
>
> I believe so; you need something to implement __await__. This is true in any
> language that implements co-routines.
>
>> Unfortunately, I can't see myself choosing asyncio in that
>> situation. Which again means that asyncio becomes "something that the
>> average user can't use". Which in turn further entrenches it as a
>> specialist-only tool.
>
> You forgot to append "... yet" to that statement. Just because something
> isn't available out of the box without some effort to support doesn't mean
> it will never happen, else there would be absolutely no Python 3 users out
> there.

Fair point. Yuri mentioned aiohttp, as well. The one difference
between this and Python 2/3, is that here you *have* to have two
separate im

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Paul Moore]

On 5 May 2015 at 21:38, Guido van Rossum  wrote:
> Jumping in to correct one fact.
>
> On Tue, May 5, 2015 at 12:44 PM, Brett Cannon  wrote:
>>
>>
>> On Tue, May 5, 2015 at 3:14 PM Paul Moore  wrote:
>>>
>>> Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
>>> but that seems to be callback-based rather than coroutine-based?
>>
>>
>> Yep.
>
>
> The run_in_executor call is not callback-based -- the confusion probably
> stems from the name of the function argument ('callback'). It actually
> returns a Future representing the result (or error) of an operation, where
> the operation is represented by the function argument. So if you have e.g. a
> function
>
> def factorial(n):
> return 1 if n <= 0 else n*factorial(n-1)
>
> you can run it in an executor from your async(io) code like this:
>
> loop = asyncio.get_event_loop()
> result = yield from loop.run_in_executor(factorial, 100)
>
> (In a PEP 492 coroutine substitute await for yield from.)

Thanks, that's an important correction. Given that, run_in_executor is
the link to blocking calls that I was searching for. And yes, the
"callback" terminology does make this far from obvious, unfortunately.
As does the point at which it's introduced (before futures have been
described) and the fact that it says "this method is a coroutine"
rather than "this method returns a Future"[1].

Paul

[1] I'm still struggling to understand the terminology, so if those
two statements are equivalent, that's not yet obvious to me.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

Jumping in to correct one fact.

On Tue, May 5, 2015 at 12:44 PM, Brett Cannon  wrote:

>
> On Tue, May 5, 2015 at 3:14 PM Paul Moore  wrote:
>
>> Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
>> but that seems to be callback-based rather than coroutine-based?
>>
>
> Yep.
>

The run_in_executor call is not callback-based -- the confusion probably
stems from the name of the function argument ('callback'). It actually
returns a Future representing the result (or error) of an operation, where
the operation is represented by the function argument. So if you have e.g.
a function

def factorial(n):
return 1 if n <= 0 else n*factorial(n-1)

you can run it in an executor from your async(io) code like this:

loop = asyncio.get_event_loop()
result = yield from loop.run_in_executor(factorial, 100)

(In a PEP 492 coroutine substitute await for yield from.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Guido van Rossum]

One small clarification:

On Tue, May 5, 2015 at 12:40 PM, Jim J. Jewett  wrote:

> [...] but I don't understand how this limitation works with things like a
> per-line file iterator that might need to wait for the file to
> be initially opened.
>

 Note that PEP 492 makes it syntactically impossible to use a coroutine
function to implement an iterator using yield; this is because the
generator machinery is needed to implement the coroutine machinery.
However, the PEP allows the creation of asynchronous iterators using
classes that implement __aiter__ and __anext__. Any blocking you need to do
can happen in either of those. You just use `async for` to iterate over
such an "asynchronous stream".

(There's an issue with actually implementing an asynchronous stream mapped
to a disk file, because I/O multiplexing primitives like select() don't
actually support waiting for disk files -- but this is an unrelated
problem, and asynchronous streams are useful to handle I/O to/from network
connections, subprocesses (pipes) or local RPC connections. Checkout the
streams  and
subprocess 
submodules of the asyncio package. These streams would be great candidates
for adding __aiter__/__anext__ to support async for-loops, so the idiom for
iterating over them can once again closely resemble the idiom for iterating
over regular (synchronous) streams using for-loops.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Tue, May 5, 2015 at 1:39 PM, Paul Moore  wrote:

> It's very hard to separate coroutines from asyncio, because there's no
> other example (not even a toy one) to reason about.
>

What about Greg Ewing's example?
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

On 2015-05-05 4:39 PM, Paul Moore wrote:

Is there anyone who feels they could write a stripped down but working
example of a valid Python event loop*without*  the asyncio aspects? Or
is that what David Beazley's talk does?

Yes, in David's talk, where he starts to use 'yield from' you
can simply use new coroutines.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Paul Moore]

On 5 May 2015 at 21:00, Yury Selivanov  wrote:
> On 2015-05-05 3:40 PM, Jim J. Jewett wrote:
>>
>> On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492.
>>
>> Where are the following over-simplifications wrong?
>>
>> (1)  The PEP is intended for use (almost exclusively) with
>> asychronous IO and a scheduler such as the asynchio event loop.
>
> Yes. You can also use it for UI loops.  Basically, anything
> that can call your code asynchronously.

Given that the stdlib doesn't provide an example of such a UI loop,
what would a 3rd party module need to implement to provide such a
thing? Can any of the non-IO related parts of asyncio be reused for
the purpose, or must the 3rd party module implement everything from
scratch?

To me, this is an important question, as it cuts directly to the heart
of the impression people have that coroutines and async are "only for
asyncio".

I'd be interested in writing, for instructional purposes, a toy but
complete event loop. But I'm *not* really interested in trying to
reverse engineer the required interface.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Tue, May 5, 2015 at 1:44 PM, Paul Moore  wrote:

> [Guido]
> > The run_in_executor call is not callback-based -- the confusion probably
> > stems from the name of the function argument ('callback'). It actually
> > returns a Future representing the result (or error) of an operation,
> where
> > the operation is represented by the function argument. So if you have
> e.g. a
> > function
> >
> > def factorial(n):
> > return 1 if n <= 0 else n*factorial(n-1)
> >
> > you can run it in an executor from your async(io) code like this:
> >
> > loop = asyncio.get_event_loop()
> > result = yield from loop.run_in_executor(factorial, 100)
> >
> > (In a PEP 492 coroutine substitute await for yield from.)
>
> Thanks, that's an important correction. Given that, run_in_executor is
> the link to blocking calls that I was searching for. And yes, the
> "callback" terminology does make this far from obvious, unfortunately.
> As does the point at which it's introduced (before futures have been
> described) and the fact that it says "this method is a coroutine"
> rather than "this method returns a Future"[1].
>
> Paul
>
> [1] I'm still struggling to understand the terminology, so if those
> two statements are equivalent, that's not yet obvious to me.
>

I apologize for the confusing documentation. We need more help from
qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
after that I was exhausted and did not want to take on writing the end user
documentation as well, so it was left unfinished. :-(

In PEP 3156 (asyncio package) there are really three separate concepts:

- Future, which is a specific class (of which Task is a subclass);

- coroutine, by which in this context is meant a generator object obtained
by calling a generator function decorated with @asyncio.coroutine and
written to conform to the asyncio protocol for coroutines (i.e. don't use
bare yield, only use yield from, and the latter always with either a Future
or a coroutine as argument);

- either of the above, which is actually the most common requirement --
most asyncio functions that support one also support the other, and either
is allowable as the argument to `yield from`.

In the implementation we so often flipped between Future and coroutine that
I imagine sometimes the implementation and docs differ; also, we don't have
a good short name for "either of the above" so we end up using one or the
other as a shorthand.

*Unless* you want to attach callbacks, inspect the result or exception, or
cancel it (all of which require a Future), your code shouldn't be concerned
about the difference -- you should just use `res = yield from func(args)`
and use try/except to catch exceptions if you care. And if you do need a
Future, you can call the function asyncio.async() on it (which in PEP 492
is renamed to ensure_future()).

In the PEP 492 world, these concepts map as follows:

- Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an
alias for Future.__iter__);

- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note
that @asyncio.coroutine incorporates the latter);

- "either of the above" maps to "awaitable".

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Paul Moore]

On 5 May 2015 at 21:57, Guido van Rossum  wrote:
> On Tue, May 5, 2015 at 1:39 PM, Paul Moore  wrote:
>>
>> It's very hard to separate coroutines from asyncio, because there's no
>> other example (not even a toy one) to reason about.
>
> What about Greg Ewing's example?
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

That doesn't cover any of the higher level abstractions like tasks or
futures (at least not by those names or with those interfaces). And I
don't see where the PEP 492 additions would fit in (OK, "replace yield
from with await" is part of it, but I don't see the rest).

We may be talking at cross purposes here. There's a lot of asyncio
that doesn't seem to me to be IO-related. Specifically the future and
task abstractions. I view those as relevant to "coroutine programming
in Python" because they are referenced in any discussion of coroutines
(you yield from a future, for example). If you see them as purely
asyncio related (and not directly usable from outside of an asyncio
context) then that may explain some of my confusion (but at the cost
of reducing the coroutine concept to something pretty trivial in the
absence of a library that independently implements these concepts).

In some ways I wish there had been an "asyncio" library that covered
the areas that are fundamentally about IO multiplexing. And a separate
library (just "async", maybe, although that's now a bad idea as it
clashes with a keyword :-)) that covered generic event loop, task and
synchronisation areas. But that's water under the bridge now.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492: Please mention the Event Loop

[Jim J. Jewett]

On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:

> It's not as
> complicated as it seems when you realize there is an event loop driving
> everything (which people have been leaving out of the conversation since it
> doesn't tie into the syntax directly).

Another reason people don't realize it is that the PEP goes out
of its way to avoid saying so.

I understand that you (and Yuri) don't want to tie the PEP too
tightly to the specific event loop implementation in
asyncio.events.AbstractEventLoop, but ... that particular
conflation isn't really what people are confused about.

"coroutines" often brings up thoughts of independent tasks.  Yuri may
well know that "(Python has asymmetric coroutines, that's it)", but
others have posted that this was a surprise -- and the people posting
here have far more python experience than most readers will.

Anyone deeply involved enough to recognize that this PEP is only about

  (1) a particular type of co-routine -- a subset even of prior python usage

  (2) used for a particular purpose

  (3) coordinated via an external scheduler

will already know that they can substitute other event loops.

Proposed second paragraph of the abstract:

This PEP assumes that the asynchronous tasks are scheduled and
coordinated by an Event Loop similar to that of stdlib module
asyncio.events.AbstractEventLoop.  While the PEP is not tied to
any specific Event Loop implementation, it is relevant only to
the kind of coroutine that uses "yield" as a signal to the scheduler,
indicating that the coroutine will be waiting until an event (such
as IO) is completed.

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sub-claasing pathlib.Path seems impossible

[Ned Deily]

In article 
,
 Christophe Bal  wrote:
> In this post
>  oes-not-work/29854141#29854141>,
> I have noticed a problem with the following code.
[...]
> This breaks the sub-classing from Python point of view. In the post
>  oes-not-work/29854141#29854141>,
> I give a hack to sub-class Path but it's a bit Unpythonic.

In article 
,
 Guido van Rossum  wrote:
> It does sound like subclassing Path should be made easier.

Christophe, if you want to pursue this, you should open an issue for it 
on the Python bug tracker, bugs.python.org.  Otherwise, it will likely 
be forgotten here.

-- 
 Ned Deily,
 n...@acm.org

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Guido van Rossum]

On Tue, May 5, 2015 at 2:01 PM, Paul Moore  wrote:

> On 5 May 2015 at 21:00, Yury Selivanov  wrote:
> > On 2015-05-05 3:40 PM, Jim J. Jewett wrote:
> >>
> >> On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated
> PEP492.
> >>
> >> Where are the following over-simplifications wrong?
> >>
> >> (1)  The PEP is intended for use (almost exclusively) with
> >> asychronous IO and a scheduler such as the asynchio event loop.
> >
> > Yes. You can also use it for UI loops.  Basically, anything
> > that can call your code asynchronously.
>
> Given that the stdlib doesn't provide an example of such a UI loop,
> what would a 3rd party module need to implement to provide such a
> thing? Can any of the non-IO related parts of asyncio be reused for
> the purpose, or must the 3rd party module implement everything from
> scratch?
>
> To me, this is an important question, as it cuts directly to the heart
> of the impression people have that coroutines and async are "only for
> asyncio".
>
> I'd be interested in writing, for instructional purposes, a toy but
> complete event loop. But I'm *not* really interested in trying to
> reverse engineer the required interface.
>

This is a great idea. What kind of application do you have in mind?

I think the main real-life use case for using coroutines with a UI event
loop is newer Windows code. C# (and IIUC VB) has coroutines very much along
the lines of PEP 492, and all code that does any kind of I/O (whether disk
or network) must be written as a coroutine. This requirement is enforced by
the C# compiler: the basic system calls for doing I/O are coroutines, and
in order to get their result you must use an await expression, which in
turn may only be used in a coroutine. Thus all code that may invoke an I/O
call ends up being a coroutine. This is exactly the type of constraint
we're trying to introduce into Python with PEP 492 (except of course we're
not making all I/O primitives coroutines -- that would be madness, we're
going with optional instead).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Paul Moore]

On 5 May 2015 at 22:12, Guido van Rossum  wrote:
> I apologize for the confusing documentation. We need more help from
> qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
> after that I was exhausted and did not want to take on writing the end user
> documentation as well, so it was left unfinished. :-(

Fair enough. When I properly document one of my projects, *then* I'll
think about complaining :-) These things happen.

> In PEP 3156 (asyncio package) there are really three separate concepts:
>
> - Future, which is a specific class (of which Task is a subclass);
>
> - coroutine, by which in this context is meant a generator object obtained
> by calling a generator function decorated with @asyncio.coroutine and
> written to conform to the asyncio protocol for coroutines (i.e. don't use
> bare yield, only use yield from, and the latter always with either a Future
> or a coroutine as argument);
>
> - either of the above, which is actually the most common requirement -- most
> asyncio functions that support one also support the other, and either is
> allowable as the argument to `yield from`.
>
> In the implementation we so often flipped between Future and coroutine that
> I imagine sometimes the implementation and docs differ; also, we don't have
> a good short name for "either of the above" so we end up using one or the
> other as a shorthand.

OK, that makes a lot of sense.

> *Unless* you want to attach callbacks, inspect the result or exception, or
> cancel it (all of which require a Future), your code shouldn't be concerned
> about the difference -- you should just use `res = yield from func(args)`
> and use try/except to catch exceptions if you care. And if you do need a
> Future, you can call the function asyncio.async() on it (which in PEP 492 is
> renamed to ensure_future()).

Again, makes sense. Although there are some bits of example code in
the docs that call asyncio.async() on a coroutine and throw away the
result (for example,
https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete).
That confuses me. Are you saying that async() modifies its (coroutine)
argument to make it a Future? Rather than wrapping a coroutine in a
Future, which gets returned?

> In the PEP 492 world, these concepts map as follows:
>
> - Future translates to "something with an __await__ method" (and asyncio
> Futures are trivially made compliant by defining Future.__await__ as an
> alias for Future.__iter__);
>
> - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
> with `async def` or a generator decorated with @types.coroutine -- note that
> @asyncio.coroutine incorporates the latter);
>
> - "either of the above" maps to "awaitable".

OK. Although "future" is a nicer term than "something with an
__await__ method" and the plethora of flavours of coroutine is not
great. But given that the only term we'll need in common cases is
"awaitable", it's still a net improvement.

So in the PEP 492 world, there's no such thing as a Task outside of
asyncio? Or, to put it another way, a Task is only relevant in an IO
context (unless an alternative event loop library implemented a
similar concept), and we should only be talking in terms of awaitables
and futures (given concurrent.futures and asyncio, I doubt you're
going to be able to stop people using "Future" for the generic term
for "something with an __await__ method" at best, and quite possibly
as equivalent to "awaitable", unfortunately).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Jim J. Jewett]

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:

> As for terminology, I view this discussion differently.  It's
> not about the technical details (Python has asymmetric
> coroutines, that's it), but rather on how to disambiguate
> coroutines implemented with generators and yield-from, from
> new 'async def' coroutines.

Not just "How?", but "Why?".

Why do they *need* to be disambiguated?

With the benefit of having recently read all that discussion
(as opposed to just the PEP), my answer is ... uh ... that
generators vs "async def" is NOT an important distinction.
What matters (as best I can tell) is:

"something using yield (or yield from) to mark execution context switches"

  vs

"other kinds of callables, including those using yield to make an iterator"


I'm not quite sure that the actual proposal even really separates them
effectively, in part because the terminology keeps suggesting other
distinctions instead.  (The glossary does help; just not enough.)


-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: Please mention the Event Loop

[Paul Moore]

On 5 May 2015 at 22:09, Jim J. Jewett  wrote:
> Proposed second paragraph of the abstract:
>
> This PEP assumes that the asynchronous tasks are scheduled and
> coordinated by an Event Loop similar to that of stdlib module
> asyncio.events.AbstractEventLoop.  While the PEP is not tied to
> any specific Event Loop implementation, it is relevant only to
> the kind of coroutine that uses "yield" as a signal to the scheduler,
> indicating that the coroutine will be waiting until an event (such
> as IO) is completed.

+1. If that's not accurate then by all means correct any mistakes in
it. But assuming it *is* accurate, it would help a lot.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Yury Selivanov]

On 2015-05-05 5:01 PM, Paul Moore wrote:

On 5 May 2015 at 21:00, Yury Selivanov  wrote:

On 2015-05-05 3:40 PM, Jim J. Jewett wrote:

On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492.

Where are the following over-simplifications wrong?

(1)  The PEP is intended for use (almost exclusively) with
asychronous IO and a scheduler such as the asynchio event loop.

Yes. You can also use it for UI loops.  Basically, anything
that can call your code asynchronously.

Given that the stdlib doesn't provide an example of such a UI loop,
what would a 3rd party module need to implement to provide such a
thing? Can any of the non-IO related parts of asyncio be reused for
the purpose, or must the 3rd party module implement everything from
scratch?


The idea is that you integrate processing of UI events to
your event loop of choice.  For instance, Twisted has
integration for QT and other libraries [1].  This way you
can easily combine async network (or OS) calls with your
UI logic to avoid "callback hell".

Quick search for something like that for asyncio revealed
this library: [2].  This small library actually re-implements
relevant low-level parts of the asyncio event loop on top of
QT primitives (another approach).

Yury

[1] http://twistedmatrix.com/trac/wiki/QTReactor
[2] https://github.com/harvimt/quamash#usage -- see first_50
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Paul Moore]

On 5 May 2015 at 22:25, Guido van Rossum  wrote:
>> I'd be interested in writing, for instructional purposes, a toy but
>> complete event loop. But I'm *not* really interested in trying to
>> reverse engineer the required interface.
>
> This is a great idea. What kind of application do you have in mind?

At this point, *all* I'm thinking of is a toy. So, an implementation
somewhat parallel to asyncio, but where the event loop just passes
control to the next task - so no IO multiplexing. Essentially Greg
Ewing's example up to, but not including, "Waiting for External
Events". And ideally I'd like to think that "Waiting for Resources"
can be omitted in favour of reusing
https://docs.python.org/3/library/asyncio-sync.html and
https://docs.python.org/3/library/asyncio-queue.html. My fear is,
however, that those parts of asyncio aren't reusable for other event
loops, and every event loop implementation has to reinvent those
wheels.

When I say "the required interface" I'm thinking in terms of "what's
needed to allow reuse of the generic parts of asyncio". If nothing of
asyncio is generic in those terms, then the exercise will be futile
(except in the negative sense of confirming that there are no reusable
async components in the stdlib).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

On 2015-05-05 5:31 PM, Jim J. Jewett wrote:

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:


>As for terminology, I view this discussion differently.  It's
>not about the technical details (Python has asymmetric
>coroutines, that's it), but rather on how to disambiguate
>coroutines implemented with generators and yield-from, from
>new 'async def' coroutines.

Not just "How?", but "Why?".

Why do they*need*  to be disambiguated?


To clearly show how one interacts with the other, to explain
how backwards compatibility is implemented, and to better
illustrate some additional (and necessary) restrictions
we put on 'async def' coroutines.  Otherwise, the PEP would
be completely unreadable :)


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Tue, May 5, 2015 at 2:29 PM, Paul Moore  wrote:

> On 5 May 2015 at 22:12, Guido van Rossum  wrote:
> > I apologize for the confusing documentation. We need more help from
> > qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
> > after that I was exhausted and did not want to take on writing the end
> user
> > documentation as well, so it was left unfinished. :-(
>
> Fair enough. When I properly document one of my projects, *then* I'll
> think about complaining :-) These things happen.
>
> > In PEP 3156 (asyncio package) there are really three separate concepts:
> >
> > - Future, which is a specific class (of which Task is a subclass);
> >
> > - coroutine, by which in this context is meant a generator object
> obtained
> > by calling a generator function decorated with @asyncio.coroutine and
> > written to conform to the asyncio protocol for coroutines (i.e. don't use
> > bare yield, only use yield from, and the latter always with either a
> Future
> > or a coroutine as argument);
> >
> > - either of the above, which is actually the most common requirement --
> most
> > asyncio functions that support one also support the other, and either is
> > allowable as the argument to `yield from`.
> >
> > In the implementation we so often flipped between Future and coroutine
> that
> > I imagine sometimes the implementation and docs differ; also, we don't
> have
> > a good short name for "either of the above" so we end up using one or the
> > other as a shorthand.
>
> OK, that makes a lot of sense.
>
> > *Unless* you want to attach callbacks, inspect the result or exception,
> or
> > cancel it (all of which require a Future), your code shouldn't be
> concerned
> > about the difference -- you should just use `res = yield from func(args)`
> > and use try/except to catch exceptions if you care. And if you do need a
> > Future, you can call the function asyncio.async() on it (which in PEP
> 492 is
> > renamed to ensure_future()).
>
> Again, makes sense. Although there are some bits of example code in
> the docs that call asyncio.async() on a coroutine and throw away the
> result (for example,
>
> https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete
> ).
> That confuses me. Are you saying that async() modifies its (coroutine)
> argument to make it a Future? Rather than wrapping a coroutine in a
> Future, which gets returned?
>

No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future
alone. I'm stumped why that example calls async() and then throws the
result away. I suspect it won't work without it (or else Victor wouldn't
have added the call) but the reason seems, um, deep. I think wrapping it in
a Task enters the generator in the event loop's queue of runnables --
otherwise the generator may well be garbage-collected without ever running.

Such complexity doesn't belong in such a simple example though.


> > In the PEP 492 world, these concepts map as follows:
> >
> > - Future translates to "something with an __await__ method" (and asyncio
> > Futures are trivially made compliant by defining Future.__await__ as an
> > alias for Future.__iter__);
> >
> > - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
> > with `async def` or a generator decorated with @types.coroutine -- note
> that
> > @asyncio.coroutine incorporates the latter);
> >
> > - "either of the above" maps to "awaitable".
>
> OK. Although "future" is a nicer term than "something with an
> __await__ method" and the plethora of flavours of coroutine is not
> great. But given that the only term we'll need in common cases is
> "awaitable", it's still a net improvement.
>
> So in the PEP 492 world, there's no such thing as a Task outside of
> asyncio? Or, to put it another way, a Task is only relevant in an IO
> context (unless an alternative event loop library implemented a
> similar concept), and we should only be talking in terms of awaitables
> and futures (given concurrent.futures and asyncio, I doubt you're
> going to be able to stop people using "Future" for the generic term
> for "something with an __await__ method" at best, and quite possibly
> as equivalent to "awaitable", unfortunately).
>

I'm not sure. But it's true that Futures and Tasks in asyncio serve the
purpose of linking the event loop (whose basic functioning is
callback-based) to coroutines (implemented by generators). The basic idea
is that when some I/O completes the event loop will call a callback
function registered for that particular I/O operation; the callback then is
actually a bound method of a Future or Task that causes the latter to be
marked as "complete" (i.e. having a result) which in turn will call other
callbacks (registered with the Future using add_done_callback()); in the
case of a Task (i.e. a special kind of Future that wraps a
generator/coroutine) this will resume the coroutine. (Actually it may
resume an entire stack of coroutines that are blocked waiting for each

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Paul Moore]

On 5 May 2015 at 22:38, Yury Selivanov  wrote:
> n 2015-05-05 5:01 PM, Paul Moore wrote:
>>
>> On 5 May 2015 at 21:00, Yury Selivanov  wrote:
>>>
>>> On 2015-05-05 3:40 PM, Jim J. Jewett wrote:

 On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated
 PEP492.

 Where are the following over-simplifications wrong?

 (1)  The PEP is intended for use (almost exclusively) with
 asychronous IO and a scheduler such as the asynchio event loop.
>>>
>>> Yes. You can also use it for UI loops.  Basically, anything
>>> that can call your code asynchronously.
>>
>> Given that the stdlib doesn't provide an example of such a UI loop,
>> what would a 3rd party module need to implement to provide such a
>> thing? Can any of the non-IO related parts of asyncio be reused for
>> the purpose, or must the 3rd party module implement everything from
>> scratch?
>
> The idea is that you integrate processing of UI events to
> your event loop of choice.  For instance, Twisted has
> integration for QT and other libraries [1].  This way you
> can easily combine async network (or OS) calls with your
> UI logic to avoid "callback hell".

We seem to be talking at cross purposes. You say the PEP is *not*
exclusively intended for use with asyncio. You mention UI loops, but
when asked how to implement such a loop, you say that I integrate UI
events into my event loop of choice. But what options do I have for
"my event loop of choice"? Please provide a concrete example that
isn't asyncio. Can I use PEP 492 with Twisted (I doubt it, as Twisted
doesn't use yield from, which is Python 3.x only)? I contend that
there *is* no concrete example that currently exists, so I'm asking
what I'd need to do to write one. You pointed at qamash, but that
seems to be subclassing asyncio, so isn't "something that isn't
asyncio".

Note that I don't have a problem with there being no existing
implementation other than asyncio. I'd just like it if we could be
clear over exactly what we mean when we say "the PEP is not tied to
asyncio". It feels like the truth currently is "you can write your own
async framework that uses the new features introduced by the PEP". I
fully expect that *if* there's a need for async frameworks that aren't
fundamentally IO multiplexors, then it'll get easier to write them
over time (the main problem right now is a lack of good tutorial
examples of how to do so). But at the moment, asyncio seems to be the
only game in town (and I can imagine that it'll always be the main IO
multiplexor, unless existing frameworks like Twisted choose to compete
rather than integrate).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Nathaniel Smith]

On May 5, 2015 2:14 PM, "Guido van Rossum"  wrote:
>
> In the PEP 492 world, these concepts map as follows:
>
> - Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an
alias for Future.__iter__);
>
> - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note
that @asyncio.coroutine incorporates the latter);
>
> - "either of the above" maps to "awaitable".

Err, aren't the first and third definitions above identical?

Surely we want to say: an async def function is a convenient shorthand for
creating a custom awaitable (exactly like how generators are a convenient
shorthand for creating custom iterators), and a Future is-an awaitable that
also adds some extra methods.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Guido van Rossum]

On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith  wrote:

On May 5, 2015 2:14 PM, "Guido van Rossum"  wrote:
> >
> > In the PEP 492 world, these concepts map as follows:
> >
> > - Future translates to "something with an __await__ method" (and asyncio
> Futures are trivially made compliant by defining Future.__await__ as an
> alias for Future.__iter__);
> >
> > - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
> with `async def` or a generator decorated with @types.coroutine -- note
> that @asyncio.coroutine incorporates the latter);
> >
> > - "either of the above" maps to "awaitable".
>
> Err, aren't the first and third definitions above identical?
>
> Surely we want to say: an async def function is a convenient shorthand for
> creating a custom awaitable (exactly like how generators are a convenient
> shorthand for creating custom iterators), and a Future is-an awaitable that
> also adds some extra methods.
>

The current PEP 492 proposal does endow the object returned by calling an
async function (let's call it a coroutine object) with an __await__ method.
And there's a good reason for this -- the bytecode generated for await
treats coroutine objects special, just like the bytecode generated for
yield-from treats generator objects special. The special behavior they have
in common is the presence of send() and throw() methods, which are used to
allow send() and throw() calls on the outer generator to be passed into the
inner generator with minimal fuss. (This is the reason why "yield from X"
is *not* equivalent to "for x in X: yield x".)

@Yury: I have a feeling the PEP could use more clarity here -- perhaps the
section "Await Expression" should explain what the interepreter does for
each type of awaitable?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Yury Selivanov]

Paul,

On 2015-05-05 5:54 PM, Paul Moore wrote:

On 5 May 2015 at 22:38, Yury Selivanov  wrote:

n 2015-05-05 5:01 PM, Paul Moore wrote:

On 5 May 2015 at 21:00, Yury Selivanov  wrote:

On 2015-05-05 3:40 PM, Jim J. Jewett wrote:

On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated
PEP492.

Where are the following over-simplifications wrong?

(1)  The PEP is intended for use (almost exclusively) with
asychronous IO and a scheduler such as the asynchio event loop.

Yes. You can also use it for UI loops.  Basically, anything
that can call your code asynchronously.

Given that the stdlib doesn't provide an example of such a UI loop,
what would a 3rd party module need to implement to provide such a
thing? Can any of the non-IO related parts of asyncio be reused for
the purpose, or must the 3rd party module implement everything from
scratch?

The idea is that you integrate processing of UI events to
your event loop of choice.  For instance, Twisted has
integration for QT and other libraries [1].  This way you
can easily combine async network (or OS) calls with your
UI logic to avoid "callback hell".

We seem to be talking at cross purposes. You say the PEP is *not*
exclusively intended for use with asyncio. You mention UI loops, but
when asked how to implement such a loop, you say that I integrate UI
events into my event loop of choice. But what options do I have for
"my event loop of choice"? Please provide a concrete example that
isn't asyncio.


Yes, there is no other popular event loop for 3.4 other
than asyncio, that uses coroutines based on generators
(as far as I know).

And yes, the PEP is not exclusively intended for use
with asyncio, but asyncio is the only library that ships
with Python, and is Python 3 ready, so its users will be
the first ones to directly benefit from this proposal.


Can I use PEP 492 with Twisted (I doubt it, as Twisted
doesn't use yield from, which is Python 3.x only)? I contend that
there *is* no concrete example that currently exists, so I'm asking
what I'd need to do to write one. You pointed at qamash, but that
seems to be subclassing asyncio, so isn't "something that isn't
asyncio".


When Twisted is ported to Python 3, I'd be really surprised
if it doesn't allow to use the new syntax.  @inlineCallbacks
implements a trampoline to make 'yields' work.  This is a
much slower approach than using 'yield from' (and 'await'
from PEP 492).  Not mentioning 'async with' and 'async for'
features.  (There shouldn't be a problem to support both
@inlineCallbacks and PEP 492 approach, if I'm not missing
something).



Note that I don't have a problem with there being no existing
implementation other than asyncio. I'd just like it if we could be
clear over exactly what we mean when we say "the PEP is not tied to
asyncio".



Well, "the PEP is not tied to asyncio" -- this is correct.
*The new syntax and new protocols know nothing about asyncio*.

asyncio will know about the PEP by implementing new protocols
where required etc (but supporting these new features isn't
in the scope of the PEP).



It feels like the truth currently is "you can write your own
async framework that uses the new features introduced by the PEP". I
fully expect that *if* there's a need for async frameworks that aren't
fundamentally IO multiplexors, then it'll get easier to write them
over time (the main problem right now is a lack of good tutorial
examples of how to do so). But at the moment, asyncio seems to be the
only game in town (and I can imagine that it'll always be the main IO
multiplexor, unless existing frameworks like Twisted choose to compete
rather than integrate).


Agree.  But if the existing frameworks choose to compete,
or someone decides to write something better than asyncio,
they can benefit from PEP 492.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Guido van Rossum]

On Tue, May 5, 2015 at 2:40 PM, Paul Moore  wrote:

> On 5 May 2015 at 22:25, Guido van Rossum  wrote:
>
[Paul:]

> >> I'd be interested in writing, for instructional purposes, a toy but
> >> complete event loop. But I'm *not* really interested in trying to
> >> reverse engineer the required interface.
> >
> > This is a great idea. What kind of application do you have in mind?
>
> At this point, *all* I'm thinking of is a toy. So, an implementation
> somewhat parallel to asyncio, but where the event loop just passes
> control to the next task - so no IO multiplexing. Essentially Greg
> Ewing's example up to, but not including, "Waiting for External
> Events". And ideally I'd like to think that "Waiting for Resources"
> can be omitted in favour of reusing
> https://docs.python.org/3/library/asyncio-sync.html and
> https://docs.python.org/3/library/asyncio-queue.html. My fear is,
> however, that those parts of asyncio aren't reusable for other event
> loops, and every event loop implementation has to reinvent those
> wheels.
>

It was never a goal of asyncio to have parts that were directly reusable by
other event loops without pulling in (almost) all of asyncio. The
interoperability offered by asyncio allows other event loops to implement
the same low-level interface as asyncio, or to build on top of asyncio.
(This is why the event loop uses callbacks and isn't coroutines/generators
all the way down.) Note that asyncio.get_event_loop() may return a loop
implemented by some other framework, and the rest of asyncio will then use
that event loop. This is enabled by the EventLoopPolicy interface.


> When I say "the required interface" I'm thinking in terms of "what's
> needed to allow reuse of the generic parts of asyncio". If nothing of
> asyncio is generic in those terms, then the exercise will be futile
> (except in the negative sense of confirming that there are no reusable
> async components in the stdlib).
>
> Paul
>

What do you hope to learn or teach by creating this toy example? And how do
you define "a complete event loop"?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: Please mention the Event Loop

[Yury Selivanov]

Jim,

On 2015-05-05 5:09 PM, Jim J. Jewett wrote:

On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:


It's not as
complicated as it seems when you realize there is an event loop driving
everything (which people have been leaving out of the conversation since it
doesn't tie into the syntax directly).

[..]

Proposed second paragraph of the abstract:

This PEP assumes that the asynchronous tasks are scheduled and
coordinated by an Event Loop similar to that of stdlib module
asyncio.events.AbstractEventLoop.  While the PEP is not tied to
any specific Event Loop implementation, it is relevant only to
the kind of coroutine that uses "yield" as a signal to the scheduler,
indicating that the coroutine will be waiting until an event (such
as IO) is completed.




Thank you for this suggestion.  I've added it to the PEP:
https://hg.python.org/peps/rev/7ac132b24f1f


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Paul Moore]

On 5 May 2015 at 23:25, Yury Selivanov  wrote:
>> Note that I don't have a problem with there being no existing
>> implementation other than asyncio. I'd just like it if we could be
>> clear over exactly what we mean when we say "the PEP is not tied to
>> asyncio".
>
> Well, "the PEP is not tied to asyncio" -- this is correct.
> *The new syntax and new protocols know nothing about asyncio*.
>
> asyncio will know about the PEP by implementing new protocols
> where required etc (but supporting these new features isn't
> in the scope of the PEP).

Thanks. That's something that may be worth explicitly noting in the
PEP (I don't recall it from when I last looked but that was a while
ago).
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Paul Moore]

On 5 May 2015 at 23:28, Guido van Rossum  wrote:
>> At this point, *all* I'm thinking of is a toy. So, an implementation
>> somewhat parallel to asyncio, but where the event loop just passes
>> control to the next task - so no IO multiplexing. Essentially Greg
>> Ewing's example up to, but not including, "Waiting for External
>> Events". And ideally I'd like to think that "Waiting for Resources"
>> can be omitted in favour of reusing
>> https://docs.python.org/3/library/asyncio-sync.html and
>> https://docs.python.org/3/library/asyncio-queue.html. My fear is,
>> however, that those parts of asyncio aren't reusable for other event
>> loops, and every event loop implementation has to reinvent those
>> wheels.
>
> It was never a goal of asyncio to have parts that were directly reusable by
> other event loops without pulling in (almost) all of asyncio. The
> interoperability offered by asyncio allows other event loops to implement
> the same low-level interface as asyncio, or to build on top of asyncio.
> (This is why the event loop uses callbacks and isn't coroutines/generators
> all the way down.) Note that asyncio.get_event_loop() may return a loop
> implemented by some other framework, and the rest of asyncio will then use
> that event loop. This is enabled by the EventLoopPolicy interface.

OK, that's an entirely fair comment. It's difficult to tell from the
docs - there's nothing obviously io-related about the task
abstraction, or the synchronisation or queue primitives. But there's
equally no reason to assume that they would work with another
implementation. As I mentioned somewhere else, maybe refactoring the
bits of asyncio that can be reused into an asynclib module would be
useful. But based on what you said, there's no reason to assume that
would be an easy job. And without another event loop implementation,
it's not obvious that there's a justification for doing so.

> What do you hope to learn or teach by creating this toy example? And how do
> you define "a complete event loop"?

Well, one thing I hope to learn, I guess, is what "a complete event
loop" consists of :-)

More broadly, I'd like to get a better feel for what methods are
fundamental to an event loop. IIRC, we had this discussion way back at
the beginning of the asyncio development when I was unclear about why
create_connection had to be an event loop method. In the asyncio
context, it has to be because the event loop needs to know when
connections get created (excuse me probably misremembering the exact
reason from back then). But conversely, it's easy to imagine an event
loop unrelated to socket IO that doesn't have a create_connection
method. On the other hand, an event loop with no call_soon method
seems unlikely. So in essence I'm thinking about what a "sensible
minimum" event loop might be. An event loop ABC, if you like.

And following on from there, what useful abstractions (tasks,
synchronisation and queue primitives) can be built on top of such a
minimal interface. Basically, that's what I'm hoping to learn - what
is fundamental (or at least generally applicable) and what is related
to the purpose of a given implementation.

I've probably got enough from this discussion to try writing up some
code and see where it leads me.

Paul

PS You mentioned that a the callback-based nature of the asyncio event
loop is to simplify interoperability with callback-based frameworks
like Twisted. I guess the above ignores the possibility of event loops
that *aren't* callback-based. Or maybe it doesn't - that's possibly
another class of methods (callback-focused ones) that maybe can be
separated into their own ABC.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Guido van Rossum]

I wonder if you could look at Tkinter for a very different view of the
world. While there are ways to integrate socket I/O with the Tcl/Tk event
loop, the typical Python app using Tkinter probably moves network I/O (if
it has any) to a separate thread, and ignores the delays of disk-based I/O
(because modern disk I/O is usually faster than the minimal event response
time -- assuming you don't have floppy disks :-).

I'm not entirely sure how you would use coroutines with Tkinter, but I
could imagine that e.g. mouse tracking code such as found in drawing apps
might be written more easily using a loop that uses await (or yield [from])
to get another event rather than as a callback function for the "mouse
move" event.

The mechanics of writing a multiplexer that receives Tkinter events and
uses them to decide which generator/coroutine to wake up might be too much
for your purpose, but it would provide a real-life example of an event loop
that's not built for network I/O.

On Tue, May 5, 2015 at 3:52 PM, Paul Moore  wrote:

> On 5 May 2015 at 23:28, Guido van Rossum  wrote:
> >> At this point, *all* I'm thinking of is a toy. So, an implementation
> >> somewhat parallel to asyncio, but where the event loop just passes
> >> control to the next task - so no IO multiplexing. Essentially Greg
> >> Ewing's example up to, but not including, "Waiting for External
> >> Events". And ideally I'd like to think that "Waiting for Resources"
> >> can be omitted in favour of reusing
> >> https://docs.python.org/3/library/asyncio-sync.html and
> >> https://docs.python.org/3/library/asyncio-queue.html. My fear is,
> >> however, that those parts of asyncio aren't reusable for other event
> >> loops, and every event loop implementation has to reinvent those
> >> wheels.
> >
> > It was never a goal of asyncio to have parts that were directly reusable
> by
> > other event loops without pulling in (almost) all of asyncio. The
> > interoperability offered by asyncio allows other event loops to implement
> > the same low-level interface as asyncio, or to build on top of asyncio.
> > (This is why the event loop uses callbacks and isn't
> coroutines/generators
> > all the way down.) Note that asyncio.get_event_loop() may return a loop
> > implemented by some other framework, and the rest of asyncio will then
> use
> > that event loop. This is enabled by the EventLoopPolicy interface.
>
> OK, that's an entirely fair comment. It's difficult to tell from the
> docs - there's nothing obviously io-related about the task
> abstraction, or the synchronisation or queue primitives. But there's
> equally no reason to assume that they would work with another
> implementation. As I mentioned somewhere else, maybe refactoring the
> bits of asyncio that can be reused into an asynclib module would be
> useful. But based on what you said, there's no reason to assume that
> would be an easy job. And without another event loop implementation,
> it's not obvious that there's a justification for doing so.
>
> > What do you hope to learn or teach by creating this toy example? And how
> do
> > you define "a complete event loop"?
>
> Well, one thing I hope to learn, I guess, is what "a complete event
> loop" consists of :-)
>
> More broadly, I'd like to get a better feel for what methods are
> fundamental to an event loop. IIRC, we had this discussion way back at
> the beginning of the asyncio development when I was unclear about why
> create_connection had to be an event loop method. In the asyncio
> context, it has to be because the event loop needs to know when
> connections get created (excuse me probably misremembering the exact
> reason from back then). But conversely, it's easy to imagine an event
> loop unrelated to socket IO that doesn't have a create_connection
> method. On the other hand, an event loop with no call_soon method
> seems unlikely. So in essence I'm thinking about what a "sensible
> minimum" event loop might be. An event loop ABC, if you like.
>
> And following on from there, what useful abstractions (tasks,
> synchronisation and queue primitives) can be built on top of such a
> minimal interface. Basically, that's what I'm hoping to learn - what
> is fundamental (or at least generally applicable) and what is related
> to the purpose of a given implementation.
>
> I've probably got enough from this discussion to try writing up some
> code and see where it leads me.
>
> Paul
>
> PS You mentioned that a the callback-based nature of the asyncio event
> loop is to simplify interoperability with callback-based frameworks
> like Twisted. I guess the above ignores the possibility of event loops
> that *aren't* callback-based. Or maybe it doesn't - that's possibly
> another class of methods (callback-focused ones) that maybe can be
> separated into their own ABC.
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/

[Python-Dev] Accepting PEP 492 (async/await)

[Guido van Rossum]

Everybody,

In order to save myself a major headache I'm hereby accepting PEP 492.

I've been following Yury's efforts carefully and I am fully confident that
we're doing the right thing here. There is only so much effort we can put
into clarifying terminology and explaining coroutines. Somebody should
write a tutorial. (I started to write one, but I ran out of time after just
describing basic yield.)

I've given Yury clear instructions to focus on how to proceed -- he's to
work with another core dev on getting the implementation ready in time for
beta 1 (scheduled for May 24, but I think the target date should be May 19).

The acceptance is provisional in the PEP 411 sense (stretching its meaning
to apply to language changes). That is, we reserve the right to change the
specification (or even withdraw it, in a worst-case scenario) until 3.6,
although I expect we won't need to do this except for some peripheral
issues (e.g. the backward compatibility flags).

I now plan to go back to PEP 484 (type hints). Fortunately in that case
there's not much *implementation* that will land (just the typing.py
module), but there's still a lot of language in the PEP that needs updating
(check the PEP 484 tracker ).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 492 (async/await)

[Guido van Rossum]

I totally forgot to publicly congratulate Yury on this PEP. He's put a huge
effort into writing the PEP and the implementation and managing the
discussion, first on python-ideas, later on python-dev. Congrats, Yury! And
thanks for your efforts. Godspeed.

On Tue, May 5, 2015 at 4:53 PM, Guido van Rossum  wrote:

> Everybody,
>
> In order to save myself a major headache I'm hereby accepting PEP 492.
>
> I've been following Yury's efforts carefully and I am fully confident that
> we're doing the right thing here. There is only so much effort we can put
> into clarifying terminology and explaining coroutines. Somebody should
> write a tutorial. (I started to write one, but I ran out of time after just
> describing basic yield.)
>
> I've given Yury clear instructions to focus on how to proceed -- he's to
> work with another core dev on getting the implementation ready in time for
> beta 1 (scheduled for May 24, but I think the target date should be May 19).
>
> The acceptance is provisional in the PEP 411 sense (stretching its meaning
> to apply to language changes). That is, we reserve the right to change the
> specification (or even withdraw it, in a worst-case scenario) until 3.6,
> although I expect we won't need to do this except for some peripheral
> issues (e.g. the backward compatibility flags).
>
> I now plan to go back to PEP 484 (type hints). Fortunately in that case
> there's not much *implementation* that will land (just the typing.py
> module), but there's still a lot of language in the PEP that needs updating
> (check the PEP 484 tracker ).
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 492 (async/await)

[Ethan Furman]

Congratulations, Yury!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 492 (async/await)

[Yury Selivanov]

On 2015-05-05 7:53 PM, Guido van Rossum wrote:

Everybody,

In order to save myself a major headache I'm hereby accepting PEP 492.

I've been following Yury's efforts carefully and I am fully confident that
we're doing the right thing here. There is only so much effort we can put
into clarifying terminology and explaining coroutines. Somebody should
write a tutorial. (I started to write one, but I ran out of time after just
describing basic yield.)




Thank you, Guido!
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 492 (async/await)

[Victor Stinner]

Hi,

2015-05-06 1:53 GMT+02:00 Guido van Rossum :
> In order to save myself a major headache I'm hereby accepting PEP 492.

Great! Congrats Yury.

> I've given Yury clear instructions to focus on how to proceed -- he's to
> work with another core dev on getting the implementation ready in time for
> beta 1 (scheduled for May 24, but I think the target date should be May 19).

The implementation takes place at:
https://bugs.python.org/issue24017

Yury works at https://github.com/1st1/cpython in git branches.

I already sent a first review. But I don't feel able to review the
change on the grammar, parser or things like that.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Terry Reedy]

On 5/5/2015 6:25 PM, Yury Selivanov wrote:


Yes, there is no other popular event loop for 3.4 other
than asyncio,


There is the tk(inter) event loop which also ships with CPython, and 
which is commonly used.



that uses coroutines based on generators


Oh ;-) Tkinter event loop is callback based.  AFAIK, so is the asyncio 
event loop, but that is somehow masked by tasks that interface to 
coroutines.  Do you think the 'somehow' could be adapted to work with 
the tkinter loop?


What I do not understand is how io events become event loop Event 
instances.  For tk, keyboard and mouse actions seen by the OS become tk 
Events associated with a widget.  Some widgets generate events. User 
code can also generate (pseudo)events.


My specific use case is to be able to run a program in a separate 
process, but display the output in the gui process -- something like 
this (in Idle, for instance).  (Apologies if this misuses the new keywords.)


async def menu_handler()
ow = OutputWindow(args)  # tk Widget
proc = subprocess.Popen (or multiprocessing equivalent)
out = (stdout from process)
await for line in out:
ow.write(line)
finish()

I want the handler to not block event processing, and disappear after 
finishing.  Might 492 make this possible someday?  Or would having 'line 
in pipe' or just 'data in pipe' translated to a tk event likely require 
a patch to tk?


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Guido van Rossum]

For this you should probably use an integration of asyncio (which can do
async subprocess output nicely) with Tkinter. Over in tulip-land there is
an demo of such an integration.

On Tue, May 5, 2015 at 6:03 PM, Terry Reedy  wrote:

> On 5/5/2015 6:25 PM, Yury Selivanov wrote:
>
>  Yes, there is no other popular event loop for 3.4 other
>> than asyncio,
>>
>
> There is the tk(inter) event loop which also ships with CPython, and which
> is commonly used.
>
>  that uses coroutines based on generators
>>
>
> Oh ;-) Tkinter event loop is callback based.  AFAIK, so is the asyncio
> event loop, but that is somehow masked by tasks that interface to
> coroutines.  Do you think the 'somehow' could be adapted to work with the
> tkinter loop?
>
> What I do not understand is how io events become event loop Event
> instances.  For tk, keyboard and mouse actions seen by the OS become tk
> Events associated with a widget.  Some widgets generate events. User code
> can also generate (pseudo)events.
>
> My specific use case is to be able to run a program in a separate process,
> but display the output in the gui process -- something like this (in Idle,
> for instance).  (Apologies if this misuses the new keywords.)
>
> async def menu_handler()
> ow = OutputWindow(args)  # tk Widget
> proc = subprocess.Popen (or multiprocessing equivalent)
> out = (stdout from process)
> await for line in out:
> ow.write(line)
> finish()
>
> I want the handler to not block event processing, and disappear after
> finishing.  Might 492 make this possible someday?  Or would having 'line in
> pipe' or just 'data in pipe' translated to a tk event likely require a
> patch to tk?
>
> --
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 5

[Ben Darnell]

On Tue, May 5, 2015 at 3:25 PM, Yury Selivanov 
wrote:

>
> Yes, there is no other popular event loop for 3.4 other
> than asyncio, that uses coroutines based on generators
> (as far as I know).
>

Tornado supports Python 3.4 and uses generator-based coroutines. We use
`yield` instead of `yield from` for compatibility with Python 2. I have a
patch to support the new async/await syntax here:
https://github.com/bdarnell/tornado/commit/e3b71c3441e9f87a29a9b112901b7644b5b6edb8

Overall, I like the PEP. I've been reluctant to embrace `yield from` for
Tornado coroutines (Tornado's Futures do not implement `__iter__`) because
I'm worried about confusion between `yield` and `yield from`, but async and
await are explicit enough that that's not really a problem.

My one request would be that there be a type or ABC corresponding to
inspect.isawaitable(). Tornado uses functools.singledispatch to handle
interoperability with other coroutine frameworks, so it would be best if we
could distinguish awaitables from other objects in a way that is compatible
with singledispatch. The patch above simply registers types.GeneratorType
which isn't quite correct.

-Ben


>
> And yes, the PEP is not exclusively intended for use
> with asyncio, but asyncio is the only library that ships
> with Python, and is Python 3 ready, so its users will be
> the first ones to directly benefit from this proposal.
>
>  Can I use PEP 492 with Twisted (I doubt it, as Twisted
>> doesn't use yield from, which is Python 3.x only)? I contend that
>> there *is* no concrete example that currently exists, so I'm asking
>> what I'd need to do to write one. You pointed at qamash, but that
>> seems to be subclassing asyncio, so isn't "something that isn't
>> asyncio".
>>
>
> When Twisted is ported to Python 3, I'd be really surprised
> if it doesn't allow to use the new syntax.  @inlineCallbacks
> implements a trampoline to make 'yields' work.  This is a
> much slower approach than using 'yield from' (and 'await'
> from PEP 492).  Not mentioning 'async with' and 'async for'
> features.  (There shouldn't be a problem to support both
> @inlineCallbacks and PEP 492 approach, if I'm not missing
> something).
>
>
>> Note that I don't have a problem with there being no existing
>> implementation other than asyncio. I'd just like it if we could be
>> clear over exactly what we mean when we say "the PEP is not tied to
>> asyncio".
>>
>
>
> Well, "the PEP is not tied to asyncio" -- this is correct.
> *The new syntax and new protocols know nothing about asyncio*.
>
> asyncio will know about the PEP by implementing new protocols
> where required etc (but supporting these new features isn't
> in the scope of the PEP).
>
>
>  It feels like the truth currently is "you can write your own
>> async framework that uses the new features introduced by the PEP". I
>> fully expect that *if* there's a need for async frameworks that aren't
>> fundamentally IO multiplexors, then it'll get easier to write them
>> over time (the main problem right now is a lack of good tutorial
>> examples of how to do so). But at the moment, asyncio seems to be the
>> only game in town (and I can imagine that it'll always be the main IO
>> multiplexor, unless existing frameworks like Twisted choose to compete
>> rather than integrate).
>>
>
> Agree.  But if the existing frameworks choose to compete,
> or someone decides to write something better than asyncio,
> they can benefit from PEP 492.
>
>
> Yury
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Greg Ewing]

Paul Moore wrote:

It would probably be helpful to have a concrete example of a basic
event loop that did *nothing* but schedule tasks. No IO waiting or
similar, just scheduling.


Take a look at the example I developed when working
on the yield-from pep:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt

The first version of the event loop presented there
does exactly that, just schedules tasks in a round-
robin fashion. Then I gradually add more features
to it.


Actually, what *is* the minimal event loop
interface that is needed for the various task/future mechanisms to
work, independently of asyncio?


I don't it's possible to answer that question,
because there isn't a single answer. The minimal
set of features that an event loop needs depends
on what you want to achieve with it.

Even the notion of "just schedules tasks" is
ambiguous. What does "schedule" mean? Does it
just mean round-robin switching between them, or
should they be able to synchronise with each
other in some way? Should it be possible for a
task to suspend itself for an interval of real
world time, or does that come under the heading
of I/O (since you're waiting for an external
event, i.e. the computer's clock reaching some
time)? Etc.

> And what features of an event loop etc
> are needed for the PEP, if it's being used outside of asyncio?)

I don't think *any* particular event loop
features are needed.

You can't pick out any of these features as
being "core". For example, it would be possible
to have an event loop that handled socket I/O
but *didn't* do round-robin scheduling -- it
could just keep on running the same task, even
if it yielded, until it blocked waiting for
an external event. Such a scheduler would
probably be quite adequate for many use cases.

It seems to me that the idea of "generator-based
tasks managed by an event loop" is more of a
design pattern than something you can write a
detailed API specification for.

Another problem with the "core" idea is that
you can't start with an event loop that "just does
scheduling" and then add on other features such
as I/O *from the outside*. There has to be some
point at which everything comes together, which
means choosing something like select() or
poll() or I/O completion queues, and build that
into the heart of your event loop. At that point
it's no longer something with a simple core.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com