Re: [Python-Dev] Unicode literals in Python 2.7

[Nick Coghlan]

On 29 April 2015 at 06:20, Adam Bartoš  wrote:
> Hello,
>
> is it possible to somehow tell Python 2.7 to compile a code entered in the
> interactive session with the flag PyCF_SOURCE_IS_UTF8 set? I'm considering
> adding support for Python 2 in my package
> (https://github.com/Drekin/win-unicode-console) and I have run into the fact
> that when u"α" is entered in the interactive session, it results in
> u"\xce\xb1" rather than u"\u03b1". As this seems to be a highly specialized
> question, I'm asking it here.

As far as I am aware, we don't have the equivalent of a "coding
cookie" for the interactive interpreter, so if anyone else knows how
to do it, I'll be learning something too :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Adam Bartoš]

This situation is a bit different from coding cookies. They are used when
we have bytes from a source file, but we don't know its encoding. During
interactive session the tokenizer always knows the encoding of the bytes. I
would think that in the case of interactive session the PyCF_SOURCE_IS_UTF8
should be always set so the bytes containing encoded non-ASCII characters
are interpreted correctly. Why I'm talking about PyCF_SOURCE_IS_UTF8?
eval(u"u'\u03b1'") -> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) ->
u'\xce\xb1'. I understand that in the second case eval has no idea how are
the given bytes encoded. But the first case is actually implemented by
encoding to utf-8 and setting PyCF_SOURCE_IS_UTF8. That's why I'm talking
about the flag.

Regards, Drekin

On Wed, Apr 29, 2015 at 9:25 AM, Nick Coghlan  wrote:

> On 29 April 2015 at 06:20, Adam Bartoš  wrote:
> > Hello,
> >
> > is it possible to somehow tell Python 2.7 to compile a code entered in
> the
> > interactive session with the flag PyCF_SOURCE_IS_UTF8 set? I'm
> considering
> > adding support for Python 2 in my package
> > (https://github.com/Drekin/win-unicode-console) and I have run into the
> fact
> > that when u"α" is entered in the interactive session, it results in
> > u"\xce\xb1" rather than u"\u03b1". As this seems to be a highly
> specialized
> > question, I'm asking it here.
>
> As far as I am aware, we don't have the equivalent of a "coding
> cookie" for the interactive interpreter, so if anyone else knows how
> to do it, I'll be learning something too :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Greg Ewing]

Yury Selivanov wrote:

Looking at the grammar -- the only downside of the current approach is that
you can't do 'await await fut'.  I still think that it reads better with
parens.  If we put 'await' to 'factor' terminal we would allow

await -fut  # await (-fut)


Is there really a need to disallow that? It would take
a fairly bizarre API to make it meaningful in the first
place, but in any case, it's fairly clear what order
of operations is intended without the parens.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Yury Selivanov wrote:


It's important to at least have 'iscoroutine' -- to check that
the object is a coroutine function.  A typical use-case would be
a web framework that lets you to bind coroutines to specific
http methods/paths:

@http.get('/spam')
async def handle_spam(request):
...

>

The other thing is that it's easy to implement this function
for CPython: just check for CO_COROUTINE flag.


But isn't that too restrictive? Any function that returns
an awaitable object would work in the above case.


One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::


I think it's a mistake that a lot of beginners may make at some
point (and in this sense it's frequent).  I really doubt that
once you were hit by it more than two times you would make it
again.


What about when you change an existing non-suspendable
function to make it suspendable, and have to deal with
the ripple-on effects of that? Seems to me that affects
everyone, not just beginners.


3. ``yield from`` does not accept coroutine objects from plain Python
generators (*not* generator-based coroutines.)

What exactly are "coroutine 
objects

from plain Python generators"?)


# *Not* decorated with @coroutine
def some_algorithm_impl():
yield 1
yield from native_coroutine() # <- this is a bug


So what you really mean is "yield-from, when used inside
a function that doesn't have @coroutine applied to it,
will not accept a coroutine object", is that right? If
so, I think this part needs re-wording, because it sounded
like you meant something quite different.

I'm not sure I like this -- it seems weird that applying
a decorator to a function should affect the semantics
of something *inside* the function -- especially a piece
of built-in syntax such as 'yield from'. It's similar
to the idea of replacing 'async def' with a decorator,
which you say you're against.

BTW, by "coroutine object", do you mean only objects
returned by an async def function, or any object having
an __await__ method? I think a lot of things would be
clearer if we could replace the term "coroutine object"
with "awaitable object" everywhere.


   ``yield from`` does not accept *native coroutine objects*
   from regular Python generators


It's the "from" there that's confusing -- it sounds
like you're talking about where the argument to
yield-from comes from, rather than where the yield-from
expression resides. In other words, we though you were
proposing to disallow *this*:

   # *Not* decorated with @coroutine
   def some_algorithm_impl():
  yield 1
  yield from iterator_implemented_by_generator()

I hope to agree that this is a perfectly legitimate
thing to do, and should remain so?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Guido van Rossum wrote:
There's a cost to __future__ imports 
too. The current proposal is a pretty clever hack -- and we've done 
similar hacks in the past


There's a benefit to having a __future__ import beyond
avoiding hackery: by turning on the __future__ you can
find out what will break when they become real keywords.

But I suppose that could be achieved by having both the
hack *and* the __future__ import available.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Greg Ewing]

Guido van Rossum wrote:

I don't care for await await x.


But do you dislike it enough to go out of your way
to disallow it?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Yury Selivanov wrote:

I also like Guido's suggestion to use "native coroutine" term.
I'll update the PEP (I have several branches of it in the repo
that I need to merge before the rename).


I'd still prefer to avoid use of the word "coroutine"
altogether as being far too overloaded.

I think even the term "native coroutine" leaves room
for ambiguity. It's not clear to me whether you intend
it to refer only to functions declared with 'async def',
or to any function that returns an awaitable object.

The term "async function" seems like a clear and
unabmigious way to refer to the former. I'm not sure
what to call the latter.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Guido van Rossum wrote:

On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov > wrote:

>

Why StopAsyncIteration?
'''

I keep wanting to propose to rename this to AsyncStopIteration.


+1, that seems more consistent to me too.


And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``


I think that's a red herring in relation to the reason
for StopAsyncIteration/AsyncStopIteration being needed.
The real reason is that StopIteration is already being
used to signal returning a value from an async function,
so it can't also be used to signal the end of an async
iteration.


One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::

@asyncio.coroutine
def useful():
asyncio.sleep(1) # this will do noting without 'yield from'

Might be useful to point out that this was the one major advantage of 
PEP 3152 -- although it wasn't enough to save that PEP, and in your 
response you pointed out that this mistake is not all that common. 
Although you seem to disagree with that here ("One of the most frequent 
mistakes ...").


I think we need some actual evidence before we can claim
that one of these mistakes is more easily made than the
other.

A priori, I would tend to assume that failing to use
'await' when it's needed would be the more insidious one.
If you mistakenly treat the return value of a function as
a future when it isn't one, you will probably find out
about it pretty quickly even under the old regime,
since most functions don't return iterators.

On the other hand, consider refactoring a function that
was previously not a coroutine so that it now is. All
existing calls to that function now need to be located
and have either 'yield from' or 'await' put in front
of them. There are three possibilities:

1. The return value is not used. The destruction-before-
iterated-over heuristic will catch this (although since
it happens in a destructor, you won't get an exception
that propagates in the usual way).

2. Some operation is immediately performed on the return
value. Most likely this will fail, so you will find out
about the problem promptly and get a stack trace,
although the error message will be somewhat tangentially
related to the cause.

3. The return value is stored away for later use. Some
time later, an operation on it will fail, but it will
no longer be obvious where the mistake was made.

So it's all a bit of a mess, IMO. But maybe it's
good enough. We need data. How often have people been
bitten by this kind of problem, and how much trouble
did it cause them?

Does send() make sense for a native coroutine? Check PEP 380. I think 
the only way to access the send() argument is by using ``yield`` but 
that's disallowed. Or is this about send() being passed to the ``yield`` 
that ultimately suspends the chain of coroutines?


That's made me think of something else. Suppose you want
to suspend execution in an 'async def' function -- how do
you do that if 'yield' is not allowed? You may need
something like the suspend() primitive that I was thinking
of adding to PEP 3152.


No implicit wrapping in Futures
---

There is a proposal to add similar mechanism to ECMAScript 7 [2]_.  A
key difference is that JavaScript "async functions" always return a
Promise. While this approach has some advantages, it also implies that
a new Promise object is created on each "async function" invocation.


I don't see how this is different from an 'async def'
function always returning an awaitable object, or a new
awaitable object being created on each 'async def'
function invocation. Sounds pretty much isomorphic to me.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Yury Selivanov wrote:


On 2015-04-28 11:59 PM, Greg wrote:


On 29/04/2015 9:49 a.m., Guido van Rossum wrote:


*But* every generator-based coroutine *must* be
decorated with `asyncio.coroutine()`.  This is
potentially a backwards incompatible change.

See below. I worry about backward compatibility. A lot. Are you saying
that asycio-based code that doesn't use @coroutine will break in 3.5?


That seems unavoidable if the goal is for 'await' to only
work on generators that are intended to implement coroutines,


Not sure what you mean by "unavoidable".


Guido is worried about existing asyncio-based code that
doesn't always decorate its generators with @coroutine.

If I understand correctly, if you have

   @coroutine
   def coro1():
  yield from coro2()

   def coro2():
  yield from ...

then coro1() would no longer work. In other words,
some currently legitimate asyncio-based code will break
under PEP 492 even if it doesn't use any PEP 492 features.

What you seem to be trying to do here is catch the
mistake of using a non-coroutine iterator as if it
were a coroutine. By "unavoidable" I mean I can't see
a way to achieve that in all possible permutations
without giving up some backward compatibility.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Yury Selivanov wrote:


3. GenObject __iter__ and __next__ raise error
*only* if it has CO_NATIVE_COROUTINE flag.  So
iter(), next(), for..in aren't supported only
for 'async def' functions (but will work ok
on asyncio generator-based coroutines)


What about new 'async def' code called by existing
code that expects to be able to use iter() or
next() on the future objects it receives?


4. 'yield from' *only* raises an error if it yields a
*coroutine with a CO_NATIVE_COROUTINE*
from a regular generator.


Won't that prevent some existing generator-based
coroutines (ones not decorated with @coroutine)
from calling ones implemented with 'async def'?

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Greg Ewing]

Yury Selivanov wrote:


It's just like unary minus ;)


I'm confused. I thought you were arguing that your
grammar is better because it's *not* just like unary
minus?

If being just like unary minus is okay, then why
not just make it so?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Yury Selivanov]

On 2015-04-29 5:13 AM, Greg Ewing wrote:

Yury Selivanov wrote:


3. GenObject __iter__ and __next__ raise error
*only* if it has CO_NATIVE_COROUTINE flag.  So
iter(), next(), for..in aren't supported only
for 'async def' functions (but will work ok
on asyncio generator-based coroutines)


What about new 'async def' code called by existing
code that expects to be able to use iter() or
next() on the future objects it receives?


4. 'yield from' *only* raises an error if it yields a
*coroutine with a CO_NATIVE_COROUTINE*
from a regular generator.


Won't that prevent some existing generator-based
coroutines (ones not decorated with @coroutine)
from calling ones implemented with 'async def'?


It would.  But that's not a backwards compatibility
issue.

Everything will work in 3.5 without a single line
change.  If you want to use new coroutines - use
them, everything will work too.

If, however, during the refactoring you've missed
several generator-based coroutines *and* they are
not decorated with @coroutine - then yes, you will
get a runtime error.  I see absolutely no problem
with that.  It's a small price to pay for a better
design.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

On 2015-04-29 5:12 AM, Greg Ewing wrote:

Yury Selivanov wrote:
Looking at the grammar -- the only downside of the current approach 
is that

you can't do 'await await fut'.  I still think that it reads better with
parens.  If we put 'await' to 'factor' terminal we would allow

await -fut  # await (-fut)


Is there really a need to disallow that? It would take
a fairly bizarre API to make it meaningful in the first
place, but in any case, it's fairly clear what order
of operations is intended without the parens.


Greg, if grammar can prevent this kind of mistakes - it should.
I like my current approach.

Yury



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Yury Selivanov]

Greg,

On 2015-04-29 5:12 AM, Greg Ewing wrote:

Yury Selivanov wrote:


It's important to at least have 'iscoroutine' -- to check that
the object is a coroutine function.  A typical use-case would be
a web framework that lets you to bind coroutines to specific
http methods/paths:

@http.get('/spam')
async def handle_spam(request):
...

>

The other thing is that it's easy to implement this function
for CPython: just check for CO_COROUTINE flag.


But isn't that too restrictive? Any function that returns
an awaitable object would work in the above case.


It's just an example. All in all, I think that we should have
full coverage of python objects in the inspect module.  There are
many possible use cases besides the one that I used -- runtime
introspection, reflection, debugging etc, where you might need
them.




One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::


I think it's a mistake that a lot of beginners may make at some
point (and in this sense it's frequent).  I really doubt that
once you were hit by it more than two times you would make it
again.


What about when you change an existing non-suspendable
function to make it suspendable, and have to deal with
the ripple-on effects of that? Seems to me that affects
everyone, not just beginners.


I've been using coroutines on a daily basis for 6 or 7 years now,
long before asyncio we had a coroutine-based framework at my firm
(yield + trampoline).  Neither I nor my colleagues had any
problems with refactoring the code.  I really try to speak from
my experience when I say that it's not that big of a problem.

Anyways, the PEP provides set_coroutine_wrapper which should solve
the problem.




3. ``yield from`` does not accept coroutine objects from plain Python
generators (*not* generator-based coroutines.)


What exactly are "coroutine objects
from plain Python generators"?)


# *Not* decorated with @coroutine
def some_algorithm_impl():
yield 1
yield from native_coroutine() # <- this is a bug


So what you really mean is "yield-from, when used inside
a function that doesn't have @coroutine applied to it,
will not accept a coroutine object", is that right? If
so, I think this part needs re-wording, because it sounded
like you meant something quite different.

I'm not sure I like this -- it seems weird that applying
a decorator to a function should affect the semantics
of something *inside* the function -- especially a piece
of built-in syntax such as 'yield from'. It's similar
to the idea of replacing 'async def' with a decorator,
which you say you're against.


This is for the transition period. We don't want to break
existing asyncio code.  But we do want coroutines to be
a separate concept from generators. It doesn't make any
sense to iterate through coroutines or to yield-from them.

We can deprecate @coroutine decorator in 3.6 or 3.7 and
at some time remove it.



BTW, by "coroutine object", do you mean only objects
returned by an async def function, or any object having
an __await__ method? I think a lot of things would be
clearer if we could replace the term "coroutine object"
with "awaitable object" everywhere.


The PEP clearly separates awaitbale from coroutine objects.

- coroutine object is returned from coroutine call.

- awaitable is either a coroutine object or an object with
__await__.

list(), tuple(), iter(), next(), for..in etc. won't work
on objects with __await__ (unless they implement __iter__).

The problem I was discussing is about specifically
'yield from' and 'coroutine object'.




   ``yield from`` does not accept *native coroutine objects*
   from regular Python generators


It's the "from" there that's confusing -- it sounds
like you're talking about where the argument to
yield-from comes from, rather than where the yield-from
expression resides. In other words, we though you were
proposing to disallow *this*:

   # *Not* decorated with @coroutine
   def some_algorithm_impl():
  yield 1
  yield from iterator_implemented_by_generator()

I hope to agree that this is a perfectly legitimate
thing to do, and should remain so?




Sure it's perfectly normal ;)  I apologize for the poor
wording.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Yury Selivanov]

Greg,

On 2015-04-29 5:13 AM, Greg Ewing wrote:

Yury Selivanov wrote:


On 2015-04-28 11:59 PM, Greg wrote:


On 29/04/2015 9:49 a.m., Guido van Rossum wrote:


*But* every generator-based coroutine *must* be
decorated with `asyncio.coroutine()`.  This is
potentially a backwards incompatible change.

See below. I worry about backward compatibility. A lot. Are you saying
that asycio-based code that doesn't use @coroutine will break in 3.5?


That seems unavoidable if the goal is for 'await' to only
work on generators that are intended to implement coroutines,


Not sure what you mean by "unavoidable".


Guido is worried about existing asyncio-based code that
doesn't always decorate its generators with @coroutine.

If I understand correctly, if you have

   @coroutine
   def coro1():
  yield from coro2()

   def coro2():
  yield from ...

then coro1() would no longer work. In other words,
some currently legitimate asyncio-based code will break
under PEP 492 even if it doesn't use any PEP 492 features.

What you seem to be trying to do here is catch the
mistake of using a non-coroutine iterator as if it
were a coroutine. By "unavoidable" I mean I can't see
a way to achieve that in all possible permutations
without giving up some backward compatibility.


Please see my reply to Guido in other thread.

Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Stephen J. Turnbull]

Yury Selivanov writes:

 > Agree. Do you think it'd be better to combine comprehensions
 > and async lambdas in one section?

No, I'd keep them separate.  For one thing, given Guido's
long-standing lack of enthusiasm for lambda, they'll need to be
separate PEPs.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Yury Selivanov]

Greg,

On 2015-04-29 5:12 AM, Greg Ewing wrote:

Guido van Rossum wrote:

On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov 
mailto:yselivanov...@gmail.com>> wrote:

>

Why StopAsyncIteration?
'''

I keep wanting to propose to rename this to AsyncStopIteration.


+1, that seems more consistent to me too.


And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``


I think that's a red herring in relation to the reason
for StopAsyncIteration/AsyncStopIteration being needed.
The real reason is that StopIteration is already being
used to signal returning a value from an async function,
so it can't also be used to signal the end of an async
iteration.



When we start thinking about generator-coroutines (the ones that
combine 'await' and 'async yield'-something), we'll have to somehow
multiplex them to the existing generator object (at least that's
one way to do it).  StopIteration is already extremely loaded
with different special meanings.

[..]


Does send() make sense for a native coroutine? Check PEP 380. I think 
the only way to access the send() argument is by using ``yield`` but 
that's disallowed. Or is this about send() being passed to the 
``yield`` that ultimately suspends the chain of coroutines?


That's made me think of something else. Suppose you want
to suspend execution in an 'async def' function -- how do
you do that if 'yield' is not allowed? You may need
something like the suspend() primitive that I was thinking
of adding to PEP 3152.


We do this in asyncio with Futures.  We never combine 'yield' and
'yield from' in a @coroutine.  We don't need 'suspend()'.

If you need suspend()-like thing in your own framework, implement
an object with an __await__ method and await on it.




No implicit wrapping in Futures
---

There is a proposal to add similar mechanism to ECMAScript 7 
[2]_.  A

key difference is that JavaScript "async functions" always return a
Promise. While this approach has some advantages, it also implies 
that

a new Promise object is created on each "async function" invocation.


I don't see how this is different from an 'async def'
function always returning an awaitable object, or a new
awaitable object being created on each 'async def'
function invocation. Sounds pretty much isomorphic to me.



Agree.  I'll try to reword that section.

Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Victor Stinner]

Le 29 avr. 2015 10:36, "Adam Bartoš"  a écrit :
> Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.

There is a simple option to get this flag: call eval() with unicode, not
with encoded bytes.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Ethan Furman]

On 04/29, Yury Selivanov wrote:
> On 2015-04-29 5:12 AM, Greg Ewing wrote:
>> Yury Selivanov wrote:

>>> Looking at the grammar -- the only downside of the current
>>> approach is that
>>> you can't do 'await await fut'.  I still think that it reads better with
>>> parens.  If we put 'await' to 'factor' terminal we would allow
>>> 
>>> await -fut  # await (-fut)
>> 
>> Is there really a need to disallow that? It would take
>> a fairly bizarre API to make it meaningful in the first
>> place, but in any case, it's fairly clear what order
>> of operations is intended without the parens.
> 
> Greg, if grammar can prevent this kind of mistakes - it should.
> I like my current approach.

That's like saying we should always put parens around the number being
raised in

  n ** x

because

 -2**4 != (-2)**4

Please do not overuse parens.  Python is not lisp, and await is not a
function, so parens should not be needed in the common case.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Ethan,

On 2015-04-29 11:29 AM, Ethan Furman wrote:

Python is not lisp, and await is not a
function, so parens should not be needed in the common case.


Which common case you mean?

Please see this table 
https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Also please see this:
https://hg.python.org/peps/rev/d048458307b7

FWIW, 'await await fut' isn't something that you likely to see in
your life; 'await -fut' is 99.999% just a bug.  I'm not sure
why Greg is pushing his Grammar idea so aggressively.

Yury

On 2015-04-29 11:33 AM, Yury Selivanov wrote:

Ethan,

On 2015-04-29 11:29 AM, Ethan Furman wrote:

Python is not lisp, and await is not a
function, so parens should not be needed in the common case.


Which common case you mean?

Please see this table 
https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression


Yury


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Adam Bartoš]

Yes, that works for eval. But I want it for code entered during an
interactive session.

>>> u'α'
u'\xce\xb1'

The tokenizer gets b"u'\xce\xb1'" by calling PyOS_Readline and it knows
it's utf-8 encoded. But the result of evaluation is u'\xce\xb1'. Because of
how eval works, I believe that it would work correctly if the
PyCF_SOURCE_IS_UTF8 was set, but it is not. That is why I'm asking if there
is a way to set it. Also, my naive thought is that it should be always set
in the case of interactive session.


On Wed, Apr 29, 2015 at 4:59 PM, Victor Stinner 
wrote:

> Le 29 avr. 2015 10:36, "Adam Bartoš"  a écrit :
> > Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.
>
> There is a simple option to get this flag: call eval() with unicode, not
> with encoded bytes.
>
> Victor
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[andrew . svetlov]

—
Sent from Mailbox


On Wednesday Apr 29, 2015 at 7:14 PM, Yury Selivanov , 
wrote:Also please see this:

https://hg.python.org/peps/rev/d048458307b7


FWIW, 'await await fut' isn't something that you likely to see in

your life; 'await -fut' is 99.999% just a bug. 
Agree.​



 I'm not sure

why Greg is pushing his Grammar idea so aggressively.


Yury


On 2015-04-29 11:33 AM, Yury Selivanov wrote:

> Ethan,

>

> On 2015-04-29 11:29 AM, Ethan Furman wrote:

>> Python is not lisp, and await is not a

>> function, so parens should not be needed in the common case.

>

> Which common case you mean?

>

> Please see this table 

> https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression

>

> Yury


___

Python-Dev mailing list

Python-Dev@python.org

https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Guido van Rossum]

I suspect the interactive session is *not* always in UTF8. It probably
depends on the keyboard mapping of your terminal emulator. I imagine in
Windows it's the current code page.

On Wed, Apr 29, 2015 at 9:19 AM, Adam Bartoš  wrote:

> Yes, that works for eval. But I want it for code entered during an
> interactive session.
>
> >>> u'α'
> u'\xce\xb1'
>
> The tokenizer gets b"u'\xce\xb1'" by calling PyOS_Readline and it knows
> it's utf-8 encoded. But the result of evaluation is u'\xce\xb1'. Because of
> how eval works, I believe that it would work correctly if the
> PyCF_SOURCE_IS_UTF8 was set, but it is not. That is why I'm asking if there
> is a way to set it. Also, my naive thought is that it should be always set
> in the case of interactive session.
>
>
> On Wed, Apr 29, 2015 at 4:59 PM, Victor Stinner 
> wrote:
>
>> Le 29 avr. 2015 10:36, "Adam Bartoš"  a écrit :
>> > Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
>> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.
>>
>> There is a simple option to get this flag: call eval() with unicode, not
>> with encoded bytes.
>>
>> Victor
>>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Ethan Furman]

On 04/29, Yury Selivanov wrote:
> On 2015-04-29 11:29 AM, Ethan Furman wrote:
>> Python is not lisp, and await is not a
>> function, so parens should not be needed in the common case.
> 
> Which common case you mean?

The common case of not wanting to write parenthesis.  ;)

Seriously, why are the parens necessary?

  await (-coro())

from the many examples above that just work, it's a mystery
why

  await -coro()

cannot also just work and be the same as the parenthesized
version.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492: What is the real goal?

[Jim J. Jewett]

On Tue Apr 28 23:49:56 CEST 2015, Guido van Rossum quoted PEP 492:

> Rationale and Goals
> ===
>
> Current Python supports implementing coroutines via generators (PEP
> 342), further enhanced by the ``yield from`` syntax introduced in PEP
> 380. This approach has a number of shortcomings:
>
> * it is easy to confuse coroutines with regular generators, since they
>   share the same syntax; async libraries often attempt to alleviate
>   this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);

So?  PEP 492 never says what coroutines *are* in a way that explains
why it matters that they are different from generators.

Do you really mean "coroutines that can be suspended while they wait
for something slow"?

As best I can guess, the difference seems to be that a "normal"
generator is using yield primarily to say:

"I'm not done; I have more values when you want them",

but an asynchronous (PEP492) coroutine is primarily saying:

"This might take a while, go ahead and do something else meanwhile."

> As shown later in this proposal, the new ``async
> with`` statement lets Python programs perform asynchronous calls when
> entering and exiting a runtime context, and the new ``async for``
> statement makes it possible to perform asynchronous calls in iterators.

Does it really permit *making* them, or does it just signal that you
will be waiting for them to finish processing anyhow, and it doesn't
need to be a busy-wait?

As nearly as I can tell, "async with" doesn't start processing the
managed block until the "asynchronous" call finishes its work -- the
only point of the async is to signal a scheduler that the task is
blocked.

Similarly, "async for" is still linearized, with each step waiting
until the previous "asynchronous" step was not merely launched, but
fully processed.  If anything, it *prevents* within-task parallelism.

> It uses the ``yield from`` implementation with an extra step of
> validating its argument.  ``await`` only accepts an *awaitable*, which
> can be one of:

What justifies this limitation?

Is there anything wrong awaiting something that eventually uses
"return" instead of "yield", if the "this might take a while" signal
is still true?  Is the problem just that the current implementation
might not take proper advantage of task-switching?

>   Objects with ``__await__`` method are called *Future-like* objects in
>   the rest of this PEP.
>
>   Also, please note that ``__aiter__`` method (see its definition
>   below) cannot be used for this purpose.  It is a different protocol,
>   and would be like using ``__iter__`` instead of ``__call__`` for
>   regular callables.
>
>   It is a ``TypeError`` if ``__await__`` returns anything but an
>   iterator.

What would be wrong if a class just did __await__ = __anext__  ?
If the problem is that the result of __await__ should be iterable,
then why isn't __await__ = __aiter__ OK?

> ``await`` keyword is defined differently from ``yield`` and ``yield
> from``.  The main difference is that *await expressions* do not require
> parentheses around them most of the times.

Does that mean

"The ``await`` keyword has slightly higher precedence than ``yield``,
so that fewer expressions require parentheses"?

> class AsyncContextManager:
> async def __aenter__(self):
> await log('entering context')

Other than the arbitrary "keyword must be there" limitations imposed
by this PEP, how is that different from:

 class AsyncContextManager:
 async def __aenter__(self):
 log('entering context')

or even:

 class AsyncContextManager:
 def __aenter__(self):
 log('entering context')

Will anything different happen when calling __aenter__ or log?
Is it that log itself now has more freedom to let other tasks run
in the middle?


> It is an error to pass a regular context manager without ``__aenter__``
> and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
> to use ``async with`` outside of a coroutine.

Why?  Does that just mean they won't take advantage of the freedom
you offered them?  Or are you concerned that they are more likely to
cooperate badly with the scheduler in practice?

> It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
> method to ``async for``.  It is a ``SyntaxError`` to use ``async for``
> outside of a coroutine.

The same questions about why -- what is the harm?

> The following code illustrates new asynchronous iteration protocol::
>
> class Cursor:
> def __init__(self):
> self.buffer = collections.deque()
>
> def _prefetch(self):
> ...
>
> async def __aiter__(self):
> return self
>
> async def __anext__(self):
> if not self.buffer:
> self.buffer = await self._prefetch()
> if not self.buffer:
> raise StopAsyncIteration
> return self.buffer.popleft()
>
> then the `

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Ethan,

On 2015-04-29 1:25 PM, Ethan Furman wrote:

cannot also just work and be the same as the parenthesized
version.

Because it does not make any sense.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Oleg Broytman]

On Wed, Apr 29, 2015 at 09:40:43AM -0700, Guido van Rossum  
wrote:
> I suspect the interactive session is *not* always in UTF8. It probably
> depends on the keyboard mapping of your terminal emulator. I imagine in
> Windows it's the current code page.

   Even worse: in w32 it can be an OEM codepage.

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Yury Selivanov]

Hi Jim,

On 2015-04-29 1:43 PM, Jim J. Jewett wrote:

On Tue Apr 28 23:49:56 CEST 2015, Guido van Rossum quoted PEP 492:


Rationale and Goals
===

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:

* it is easy to confuse coroutines with regular generators, since they
   share the same syntax; async libraries often attempt to alleviate
   this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);

So?  PEP 492 never says what coroutines *are* in a way that explains
why it matters that they are different from generators.

Do you really mean "coroutines that can be suspended while they wait
for something slow"?

As best I can guess, the difference seems to be that a "normal"
generator is using yield primarily to say:

 "I'm not done; I have more values when you want them",

but an asynchronous (PEP492) coroutine is primarily saying:

 "This might take a while, go ahead and do something else meanwhile."


Correct.




As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.

Does it really permit *making* them, or does it just signal that you
will be waiting for them to finish processing anyhow, and it doesn't
need to be a busy-wait?


I does.



As nearly as I can tell, "async with" doesn't start processing the
managed block until the "asynchronous" call finishes its work -- the
only point of the async is to signal a scheduler that the task is
blocked.


Right.



Similarly, "async for" is still linearized, with each step waiting
until the previous "asynchronous" step was not merely launched, but
fully processed.  If anything, it *prevents* within-task parallelism.


It enables cooperative parallelism.




It uses the ``yield from`` implementation with an extra step of
validating its argument.  ``await`` only accepts an *awaitable*, which
can be one of:

What justifies this limitation?


We want to avoid people passing regular generators and random
objects to 'await', because it is a bug.



Is there anything wrong awaiting something that eventually uses
"return" instead of "yield", if the "this might take a while" signal
is still true?


If it's an 'async def' then sure, you can use it in await.


Is the problem just that the current implementation
might not take proper advantage of task-switching?


   Objects with ``__await__`` method are called *Future-like* objects in
   the rest of this PEP.

   Also, please note that ``__aiter__`` method (see its definition
   below) cannot be used for this purpose.  It is a different protocol,
   and would be like using ``__iter__`` instead of ``__call__`` for
   regular callables.

   It is a ``TypeError`` if ``__await__`` returns anything but an
   iterator.

What would be wrong if a class just did __await__ = __anext__  ?
If the problem is that the result of __await__ should be iterable,
then why isn't __await__ = __aiter__ OK?


For coroutines in PEP 492:

__await__ = __anext__ is the same as __call__ = __next__
__await__ = __aiter__ is the same as __call__ = __iter__




``await`` keyword is defined differently from ``yield`` and ``yield
from``.  The main difference is that *await expressions* do not require
parentheses around them most of the times.

Does that mean

"The ``await`` keyword has slightly higher precedence than ``yield``,
so that fewer expressions require parentheses"?


 class AsyncContextManager:
 async def __aenter__(self):
 await log('entering context')

Other than the arbitrary "keyword must be there" limitations imposed
by this PEP, how is that different from:

  class AsyncContextManager:
  async def __aenter__(self):
  log('entering context')


This is OK. The point is that you can use 'await log' in
__aenter__.  If you don't need awaits in __aenter__ you can
use them in __aexit__. If you don't need them there too,
then just define a regular context manager.



or even:

  class AsyncContextManager:
  def __aenter__(self):
  log('entering context')

Will anything different happen when calling __aenter__ or log?
Is it that log itself now has more freedom to let other tasks run
in the middle?


__aenter__ must return an awaitable.





It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
to use ``async with`` outside of a coroutine.

Why?  Does that just mean they won't take advantage of the freedom
you offered them?


Not sure I understand the question.

It doesn't make any sense in using 'async with' outside of a
coroutine.  The interpeter won't know what to do with them:
you need an event loop for that.


Or are you concern

Re: [Python-Dev] PEP 492: What is the real goal?

[Paul Moore]

On 29 April 2015 at 18:43, Jim J. Jewett  wrote:
>> Rationale and Goals
>> ===
>>
>> Current Python supports implementing coroutines via generators (PEP
>> 342), further enhanced by the ``yield from`` syntax introduced in PEP
>> 380. This approach has a number of shortcomings:
>>
>> * it is easy to confuse coroutines with regular generators, since they
>>   share the same syntax; async libraries often attempt to alleviate
>>   this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);
>
> So?  PEP 492 never says what coroutines *are* in a way that explains
> why it matters that they are different from generators.

I agree. While I don't use coroutines/asyncio, and I may never do so,
I will say that I find Python's approach very difficult to understand.

I'd hope that the point of PEP 492, by making await/async first class
language constructs, would be to make async programming more
accessible in Python. Whether that will actually be the case isn't
particularly clear to me. And whether "async programming" and
"coroutines" are the same thing, I'm even less sure of. I haven't
really followed the discussions here, because they seem to be about
details that are completely confusing to me.

In principle, I support the PEP, on the basis that working towards
better coroutine/async support in Python seems worthwhile to me. But
until the whole area is made more accessible to the average
programmer, I doubt any of this will be more than a niche area in
Python.

For example, the PEP says:

"""
New Coroutine Declaration Syntax

The following new syntax is used to declare a coroutine:

async def read_data(db):
pass
"""

Looking at the Wikipedia article on coroutines, I see an example of
how a producer/consumer process might be written with coroutines:

var q := new queue

coroutine produce
loop
while q is not full
create some new items
add the items to q
yield to consume

coroutine consume
loop
while q is not empty
remove some items from q
use the items
yield to produce

(To start everything off, you'd just run "produce").

I can't even see how to relate that to PEP 429 syntax. I'm not allowed
to use "yield", so should I use "await consume" in produce (and vice
versa)? I'd actually expect to just write 2 generators in Python, and
use .send() somehow (it's clunky and I can never remember how to write
the calls, but that's OK, it just means that coroutines don't have
first-class syntax support in Python). This is totally unrelated to
asyncio, which is the core use case for all of Python's async support.
But it's what I think of when I see the word "coroutine" (and
Wikipedia agrees).

Searching for "Async await" gets me to the Microsoft page
"Asynchronous Programming with Async and Await" describing the C#
keywords. That looks more like what PEP 429 is talking about, but it
uses the name "async method". Maybe that's what PEP should do, too,
and leave the word "coroutine" for the yielding of control that I
quoted from Wikipedia above.


Confusedly,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Ethan Furman]

On 04/29, Yury Selivanov wrote:
> On 2015-04-29 1:25 PM, Ethan Furman wrote:
>> cannot also just work and be the same as the parenthesized
>> version.
>
> Because it does not make any sense.

I obviously don't understand your position that "it does not make
any sense" -- perhaps you could explain a bit?

What I see is a suspension point that is waiting for the results of
coro(), which will be negated (and returned/assigned/whatever).
What part of that doesn't make sense?

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Yury Selivanov]

Hi Paul,

On 2015-04-29 2:26 PM, Paul Moore wrote:

On 29 April 2015 at 18:43, Jim J. Jewett  wrote:

Rationale and Goals
===

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:

* it is easy to confuse coroutines with regular generators, since they
   share the same syntax; async libraries often attempt to alleviate
   this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);

So?  PEP 492 never says what coroutines *are* in a way that explains
why it matters that they are different from generators.

I agree. While I don't use coroutines/asyncio, and I may never do so,
I will say that I find Python's approach very difficult to understand.

I'd hope that the point of PEP 492, by making await/async first class
language constructs, would be to make async programming more
accessible in Python. Whether that will actually be the case isn't
particularly clear to me. And whether "async programming" and
"coroutines" are the same thing, I'm even less sure of. I haven't
really followed the discussions here, because they seem to be about
details that are completely confusing to me.


It will make it more accessible in Python.  asyncio is getting
a lot of traction, and with this PEP accepted I can see it
only becoming easier to work with it (or any other async
frameworks that start using the new syntax/protocols).



In principle, I support the PEP, on the basis that working towards
better coroutine/async support in Python seems worthwhile to me. But
until the whole area is made more accessible to the average
programmer, I doubt any of this will be more than a niche area in
Python.

For example, the PEP says:

"""
New Coroutine Declaration Syntax

The following new syntax is used to declare a coroutine:

async def read_data(db):
 pass
"""

Looking at the Wikipedia article on coroutines, I see an example of
how a producer/consumer process might be written with coroutines:

var q := new queue

coroutine produce
 loop
 while q is not full
 create some new items
 add the items to q
 yield to consume

coroutine consume
 loop
 while q is not empty
 remove some items from q
 use the items
 yield to produce

(To start everything off, you'd just run "produce").

I can't even see how to relate that to PEP 429 syntax. I'm not allowed
to use "yield", so should I use "await consume" in produce (and vice
versa)? I'd actually expect to just write 2 generators in Python, and
use .send() somehow (it's clunky and I can never remember how to write
the calls, but that's OK, it just means that coroutines don't have
first-class syntax support in Python). This is totally unrelated to
asyncio, which is the core use case for all of Python's async support.
But it's what I think of when I see the word "coroutine" (and
Wikipedia agrees).


That Wikipedia page is very generic, and the pseudo-code
that it uses does indeed look confusing.

Here's how it might look like (this is the same pseudo-code
but tailored for PEP 492, not a real something)

  q = asyncio.Queue(maxsize=100)

  async def produce():
  # you might want to wrap it all in 'while True'

  while not q.full():
  item = create_item()
  await q.put(item)

  async def consume():
  while not q.empty():
  item = await q.get()
  process_item(item)


Thanks!
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Paul Moore]

On 29 April 2015 at 19:32, Ethan Furman  wrote:
> On 04/29, Yury Selivanov wrote:
>> On 2015-04-29 1:25 PM, Ethan Furman wrote:
>>> cannot also just work and be the same as the parenthesized
>>> version.
>>
>> Because it does not make any sense.
>
> I obviously don't understand your position that "it does not make
> any sense" -- perhaps you could explain a bit?
>
> What I see is a suspension point that is waiting for the results of
> coro(), which will be negated (and returned/assigned/whatever).
> What part of that doesn't make sense?

Would that not be "-await coro()"? What "await -coro()" would mean is
to call coro() (which would return something you can wait on), then
apply - to that (then waiting on the result of that negation). But
what does negating an awaitable object mean? Obviously you can
*define* it to mean something, but you probably didn't.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Hi Ethan,

On 2015-04-29 2:32 PM, Ethan Furman wrote:

On 04/29, Yury Selivanov wrote:

On 2015-04-29 1:25 PM, Ethan Furman wrote:

cannot also just work and be the same as the parenthesized
version.

Because it does not make any sense.

I obviously don't understand your position that "it does not make
any sense" -- perhaps you could explain a bit?

What I see is a suspension point that is waiting for the results of
coro(), which will be negated (and returned/assigned/whatever).
What part of that doesn't make sense?



Because you want operators to be resolved in the
order you see them, generally.

You want '(await -fut)' to:

1. Suspend on fut;
2. Get the result;
3. Negate it.

This is a non-obvious thing. I would myself interpret it
as:

1. Get fut.__neg__();
2. await on it.

So I want to make this syntactically incorrect:

'await -fut' would throw a SyntaxError. To do what you
want, write a pythonic '- await fut'.


Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Nathaniel Smith]

On Apr 29, 2015 11:49 AM, "Yury Selivanov"  wrote:
>
> Hi Ethan,
>
>
> On 2015-04-29 2:32 PM, Ethan Furman wrote:
>>
>> On 04/29, Yury Selivanov wrote:
>>>
>>> On 2015-04-29 1:25 PM, Ethan Furman wrote:

 cannot also just work and be the same as the parenthesized
 version.
>>>
>>> Because it does not make any sense.
>>
>> I obviously don't understand your position that "it does not make
>> any sense" -- perhaps you could explain a bit?
>>
>> What I see is a suspension point that is waiting for the results of
>> coro(), which will be negated (and returned/assigned/whatever).
>> What part of that doesn't make sense?
>>
>
> Because you want operators to be resolved in the
> order you see them, generally.
>
> You want '(await -fut)' to:
>
> 1. Suspend on fut;
> 2. Get the result;
> 3. Negate it.
>
> This is a non-obvious thing. I would myself interpret it
> as:
>
> 1. Get fut.__neg__();
> 2. await on it.
>
> So I want to make this syntactically incorrect:

As a bystander, I don't really care either way about whether await -fut is
syntactically valid (since like you say it's semantically nonsense
regardless and no one will ever write it). But I would rather like to
actually know what the syntax actually is, not just have a list of examples
(which kinda gives me perl flashbacks). Is there any simple way to state
what the rules for parsing await are? Or do I just have to read the parser
code if I want to know that?

(I suspect this may also be the impetus behind Greg's request that it just
be treated the same as unary minus. IMHO it matters much more that the
rules be predictable and teachable than that they allow or disallow every
weird edge case in exactly the right way.)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Adam Bartoš]

I am in Windows and my terminal isn't utf-8 at the beginning, but I install
custom sys.std* objects at runtime and I also install custom readline hook,
so the interactive loop gets the input from my stream objects via
PyOS_Readline. So when I enter u'α', the tokenizer gets b"u'\xce\xb1'",
which is the string encoded in utf-8, and sys.stdin.encoding == 'utf-8'.
However, the input is then interpreted as u'\xce\xb1' instead of u'\u03b1'.

On Wed, Apr 29, 2015 at 6:40 PM, Guido van Rossum  wrote:

> I suspect the interactive session is *not* always in UTF8. It probably
> depends on the keyboard mapping of your terminal emulator. I imagine in
> Windows it's the current code page.
>
> On Wed, Apr 29, 2015 at 9:19 AM, Adam Bartoš  wrote:
>
>> Yes, that works for eval. But I want it for code entered during an
>> interactive session.
>>
>> >>> u'α'
>> u'\xce\xb1'
>>
>> The tokenizer gets b"u'\xce\xb1'" by calling PyOS_Readline and it knows
>> it's utf-8 encoded. But the result of evaluation is u'\xce\xb1'. Because of
>> how eval works, I believe that it would work correctly if the
>> PyCF_SOURCE_IS_UTF8 was set, but it is not. That is why I'm asking if there
>> is a way to set it. Also, my naive thought is that it should be always set
>> in the case of interactive session.
>>
>>
>> On Wed, Apr 29, 2015 at 4:59 PM, Victor Stinner > > wrote:
>>
>>> Le 29 avr. 2015 10:36, "Adam Bartoš"  a écrit :
>>> > Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
>>> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.
>>>
>>> There is a simple option to get this flag: call eval() with unicode, not
>>> with encoded bytes.
>>>
>>> Victor
>>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Paul Moore]

On 29 April 2015 at 19:42, Yury Selivanov  wrote:
> Here's how it might look like (this is the same pseudo-code
> but tailored for PEP 492, not a real something)
>
>   q = asyncio.Queue(maxsize=100)
>
>   async def produce():
>   # you might want to wrap it all in 'while True'

I think the "loop" in the Wikipedia pseudocode was intended to be the
"while True" here, not part of the "while" on the next line.

>
>   while not q.full():
>   item = create_item()
>   await q.put(item)
>
>   async def consume():
>   while not q.empty():
>   item = await q.get()
>   process_item(item)

Thanks for that. That does look pretty OK. One question, though - it
uses an asyncio Queue. The original code would work just as well with
a list, or more accurately, something that wasn't designed for async
use. So the translation isn't completely equivalent. Also, can I run
the produce/consume just by calling produce()? My impression is that
with asyncio I need an event loop - which "traditional" coroutines
don't need. Nevertheless, the details aren't so important, it was only
a toy example anyway.

However, just to make my point precise, here's a more or less direct
translation of the Wikipedia code into Python. It doesn't actually
work, because getting the right combinations of yield and send stuff
is confusing to me. Specifically, I suspect that "yield
produce.send(None)" isn't the right way to translate "yield to
produce". But it gives the idea.

data = [1,2,3,4,5,6,7,8,9,10]

q = []

def produce():
while True:
while len(q) < 10:
if not data:
return
item = data.pop()
print("In produce - got", item)
q.append(item)
yield consume.send(None)

total = 0
def consume():
while True:
while q:
item = q.pop()
print("In consume - handling", item)
global total
total += item
yield produce.send(None)

# Prime the coroutines
produce = produce()
consume = consume()
next(produce)
print(total)

The *only* bits of this that are related to coroutines are:

1. yield consume.send(None) (and the same for produce)
2. produce = produce() (and the same for consume) priming the coroutines
3. next(produce) to start the coroutines

I don't think this is at all related to PEP 492 (which is about async)
but it's what is traditionally meant by coroutines. It would be nice
to have a simpler syntax for these "traditional" coroutines, but it's
a very niche requirement, and probably not worth it.

But the use of "coroutine" in PEP 492 for the functions introduced by
"async def" is confusing - at least to me - because I think of the
above, and not of async. Why not just call them "async functions" and
leave the term coroutine for the above flow control construct, which
is where it originated?

But maybe that ship has long sailed - the term "coroutine" is pretty
entrenched in the asyncio documentation. If so, then I guess we have
to live with the consequences.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Hi Nathaniel,

BTW, I'm going to reply to you in the other thread about
context-local objects soon.  I have some thoughts on the
topic.

On 2015-04-29 3:14 PM, Nathaniel Smith wrote:

On Apr 29, 2015 11:49 AM, "Yury Selivanov"  wrote:

Hi Ethan,


On 2015-04-29 2:32 PM, Ethan Furman wrote:

On 04/29, Yury Selivanov wrote:

On 2015-04-29 1:25 PM, Ethan Furman wrote:

cannot also just work and be the same as the parenthesized
version.

Because it does not make any sense.

I obviously don't understand your position that "it does not make
any sense" -- perhaps you could explain a bit?

What I see is a suspension point that is waiting for the results of
coro(), which will be negated (and returned/assigned/whatever).
What part of that doesn't make sense?


Because you want operators to be resolved in the
order you see them, generally.

You want '(await -fut)' to:

1. Suspend on fut;
2. Get the result;
3. Negate it.

This is a non-obvious thing. I would myself interpret it
as:

1. Get fut.__neg__();
2. await on it.

So I want to make this syntactically incorrect:

As a bystander, I don't really care either way about whether await -fut is
syntactically valid (since like you say it's semantically nonsense
regardless and no one will ever write it). But I would rather like to
actually know what the syntax actually is, not just have a list of examples
(which kinda gives me perl flashbacks). Is there any simple way to state
what the rules for parsing await are? Or do I just have to read the parser
code if I want to know that?


There is a summary of grammar changes in the PEP:
https://www.python.org/dev/peps/pep-0492/#grammar-updates

You can also see the actual grammar file from the reference
implementation:
https://github.com/1st1/cpython/blob/await/Grammar/Grammar



Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Yury Selivanov]

Paul,

On 2015-04-29 3:19 PM, Paul Moore wrote:

On 29 April 2015 at 19:42, Yury Selivanov  wrote:

Here's how it might look like (this is the same pseudo-code
but tailored for PEP 492, not a real something)

   q = asyncio.Queue(maxsize=100)

   async def produce():
   # you might want to wrap it all in 'while True'

I think the "loop" in the Wikipedia pseudocode was intended to be the
"while True" here, not part of the "while" on the next line.


   while not q.full():
   item = create_item()
   await q.put(item)

   async def consume():
   while not q.empty():
   item = await q.get()
   process_item(item)

Thanks for that. That does look pretty OK. One question, though - it
uses an asyncio Queue. The original code would work just as well with
a list, or more accurately, something that wasn't designed for async
use. So the translation isn't completely equivalent. Also, can I run
the produce/consume just by calling produce()? My impression is that
with asyncio I need an event loop - which "traditional" coroutines
don't need. Nevertheless, the details aren't so important, it was only
a toy example anyway.


Well, yes.  Coroutine is a generic term.  And you can
use PEP 492 coroutines without asyncio, in fact that's
how most tests for the reference implementation is written.

Coroutine objects have .send(), .throw() and .close() methods
(same as generator objects in Python).  You can work with them
without a loop, but loop implementations contain a lot of
logic to implement the actual cooperative execution.

You can use generators as coroutines, and nothing would
prevent you from doing that after PEP 492, moreover, for
some use-cases it might be quite a good decision.  But a
lot of the code -- web frameworks, network applications,
etc will hugely benefit from the proposal, streamlined
syntax and async for/with statements.

[..]


But the use of "coroutine" in PEP 492 for the functions introduced by
"async def" is confusing - at least to me - because I think of the
above, and not of async. Why not just call them "async functions" and
leave the term coroutine for the above flow control construct, which
is where it originated?

But maybe that ship has long sailed - the term "coroutine" is pretty
entrenched in the asyncio documentation. If so, then I guess we have
to live with the consequences.


Everybody is pulling me in a different direction :)
Guido proposed to call them "native coroutines".  Some people
think that "async functions" is a better name.  Greg loves
his "cofunction" term.

I'm flexible about how we name 'async def' functions.  I like
to call them "coroutines", because that's what they are, and
that's how asyncio calls them.  It's also convenient to use
'coroutine-object' to explain what is the result of calling
a coroutine.

Anyways, I'd be OK to start using a new term, if "coroutine" is
confusing.


Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Skip Montanaro]

On Wed, Apr 29, 2015 at 2:42 PM, Yury Selivanov 
wrote:

> Anyways, I'd be OK to start using a new term, if "coroutine" is
> confusing.
>

According to Wikipedia , term
"coroutine" was first coined in 1958, so several generations of computer
science graduates will be familiar with the textbook definition. If your
use of "coroutine" matches the textbook definition of the term, I think you
should continue to use it instead of inventing new names which will just
confuse people new to Python.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Nathaniel Smith]

On Wed, Apr 29, 2015 at 1:14 PM, Skip Montanaro
 wrote:
>
> On Wed, Apr 29, 2015 at 2:42 PM, Yury Selivanov 
> wrote:
>>
>> Anyways, I'd be OK to start using a new term, if "coroutine" is
>> confusing.
>
>
> According to Wikipedia, term "coroutine" was first coined in 1958, so
> several generations of computer science graduates will be familiar with the
> textbook definition. If your use of "coroutine" matches the textbook
> definition of the term, I think you should continue to use it instead of
> inventing new names which will just confuse people new to Python.

IIUC the problem is that Python has or will have a number of different
things that count as coroutines by that classic CS definition,
including generators, "async def" functions, and in general any object
that implements the same set of methods as one or both of these
objects, or possibly inherits from a certain abstract base class. It
would be useful to have some terms to refer specifically to async def
functions and the await protocol as opposed to generators and the
iterator protocol, and "coroutine" does not make this distinction.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Guido van Rossum]

Maybe it would help to refer to PEP 342, which first formally introduced
the concept of coroutines (as a specific use case of generators) in Python.
Personally I don't care too much which term the PEP uses, as logn as it
defines its terms. The motivation is already clear to me; it's the details
that I care about before approving this PEP.

On Wed, Apr 29, 2015 at 1:19 PM, Nathaniel Smith  wrote:

> On Wed, Apr 29, 2015 at 1:14 PM, Skip Montanaro
>  wrote:
> >
> > On Wed, Apr 29, 2015 at 2:42 PM, Yury Selivanov  >
> > wrote:
> >>
> >> Anyways, I'd be OK to start using a new term, if "coroutine" is
> >> confusing.
> >
> >
> > According to Wikipedia, term "coroutine" was first coined in 1958, so
> > several generations of computer science graduates will be familiar with
> the
> > textbook definition. If your use of "coroutine" matches the textbook
> > definition of the term, I think you should continue to use it instead of
> > inventing new names which will just confuse people new to Python.
>
> IIUC the problem is that Python has or will have a number of different
> things that count as coroutines by that classic CS definition,
> including generators, "async def" functions, and in general any object
> that implements the same set of methods as one or both of these
> objects, or possibly inherits from a certain abstract base class. It
> would be useful to have some terms to refer specifically to async def
> functions and the await protocol as opposed to generators and the
> iterator protocol, and "coroutine" does not make this distinction.
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Paul Moore]

On 29 April 2015 at 20:42, Yury Selivanov  wrote:
> Everybody is pulling me in a different direction :)

Sorry :-)

> Guido proposed to call them "native coroutines".  Some people
> think that "async functions" is a better name.  Greg loves
> his "cofunction" term.

If it helps, ignore my opinion - I'm not a heavy user of coroutines or
asyncio, so my view shouldn't have too much weight.

Thanks for your response - my question was a little off-topic, but
your reply has made things clearer for me.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

On 2015-04-29 3:24 PM, Isaac Schwabacher wrote:

On 15-04-29, Yury Selivanov  wrote:

Hi Ethan,

[..]

So I want to make this syntactically incorrect:

Does this need to be a syntax error? -"hello" raises TypeError because str doesn't have a 
__neg__, but there's no reason a str subclass couldn't define one. "TypeError: bad operand 
type for unary -: 'asyncio.Future'" is enough to clear up any misunderstandings, and if 
someone approaching a new language construct doesn't test their code well enough to at least 
execute all the code paths, the difference between a compile-time SyntaxError and a run-time 
TypeError is not going to save them.




The grammar of the language should match the most common
use case.

FWIW, I've just updated the pep with a precedence table:
https://hg.python.org/peps/rev/d355918bc0d7

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Greg Ewing]

Yury Selivanov wrote:


Won't that prevent some existing generator-based
coroutines (ones not decorated with @coroutine)
from calling ones implemented with 'async def'?


It would.  But that's not a backwards compatibility
issue.


It seems to go against Guido's desire for the new
way to be a 100% drop-in replacement for the old
way. There are various ways that old code can end
up calling new code -- subclassing, callbacks,
etc.

It also means that if person A writes a library
in the new style, then person B can't make use
of it without upgrading all of their code to the
new style as well. The new style will thus be
"infectious" in a sense.

I suppose it's up to Guido to decide whether it's
a good or bad infection. But the same kind of
reasoning seemed to be at least partly behind
the rejection of PEP 3152.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Greg Ewing]

Yury Selivanov wrote:

I'm not sure
why Greg is pushing his Grammar idea so aggressively.


Because I believe that any extra complexity in the grammar
needs a very strong justification. It's complexity in the
core language, like a new keyword, so it puts a burden on
everyone's brain.

Saying "I don't think anyone would ever need to write this,
therefore we should disallow it" is not enough, given that
there is a substantial cost to disallowing it.

If you don't think there's a cost, consider that we *both*
seem to be having trouble predicting the consequences of
your proposed syntax, and you're the one who invented it.
That's not a good sign!

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Greg,

On 2015-04-29 6:46 PM, Greg Ewing wrote:

Yury Selivanov wrote:

I'm not sure
why Greg is pushing his Grammar idea so aggressively.


Because I believe that any extra complexity in the grammar
needs a very strong justification. It's complexity in the
core language, like a new keyword, so it puts a burden on
everyone's brain.

Saying "I don't think anyone would ever need to write this,
therefore we should disallow it" is not enough, given that
there is a substantial cost to disallowing it.

If you don't think there's a cost, consider that we *both*
seem to be having trouble predicting the consequences of
your proposed syntax, and you're the one who invented it.
That's not a good sign!


Sorry, but I'm not sure where & when I had any troubles
predicting the consequences..

Please take a look at the updated PEP.  There is a
precedence table there + motivation.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; v3

[Yury Selivanov]

Greg,

On 2015-04-29 6:46 PM, Greg Ewing wrote:

Yury Selivanov wrote:


Won't that prevent some existing generator-based
coroutines (ones not decorated with @coroutine)
from calling ones implemented with 'async def'?


It would.  But that's not a backwards compatibility
issue.


It seems to go against Guido's desire for the new
way to be a 100% drop-in replacement for the old
way. There are various ways that old code can end
up calling new code -- subclassing, callbacks,
etc.

It also means that if person A writes a library
in the new style, then person B can't make use
of it without upgrading all of their code to the
new style as well. The new style will thus be
"infectious" in a sense.

I suppose it's up to Guido to decide whether it's
a good or bad infection. But the same kind of
reasoning seemed to be at least partly behind
the rejection of PEP 3152.



It's a drop-in replacement ;)  If you run your
existing code - it will 100% work just fine.

There is a probability that *when* you start applying
new syntax something could go wrong -- you're right
here.

I'm updating the PEP to explain this clearly,
and let's see what Guido thinks about that.

My opinion is that this is a solvable problem with
a clear guidelines on how to transition existing
code to the new style.

Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Guido van Rossum]

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing 
wrote:

> Yury Selivanov wrote:
>
>> I'm not sure
>> why Greg is pushing his Grammar idea so aggressively.
>>
>
> Because I believe that any extra complexity in the grammar
> needs a very strong justification. It's complexity in the
> core language, like a new keyword, so it puts a burden on
> everyone's brain.
>
> Saying "I don't think anyone would ever need to write this,
> therefore we should disallow it" is not enough, given that
> there is a substantial cost to disallowing it.
>
> If you don't think there's a cost, consider that we *both*
> seem to be having trouble predicting the consequences of
> your proposed syntax, and you're the one who invented it.
> That's not a good sign!
>

I have a slightly different view. A bunch of things *must* work, e.g.
f(await g(), await h()) or with await f(): (there's a longer list in the
PEP). Other things may be ambiguous to most readers, e.g. what does await
f() + g() mean, or can we say await await f(), and the solution is to
recommend adding parentheses that make things clear to the parser *and*
humans. Yury's proposal satisfies my requirements, and if we really find
some unpleasant edge case we can fix it during the 3.5 release (the PEP
will be provisional).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Nathaniel Smith]

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing  wrote:
> Yury Selivanov wrote:
>>
>> I'm not sure
>> why Greg is pushing his Grammar idea so aggressively.
>
>
> Because I believe that any extra complexity in the grammar
> needs a very strong justification. It's complexity in the
> core language, like a new keyword, so it puts a burden on
> everyone's brain.
>
> Saying "I don't think anyone would ever need to write this,
> therefore we should disallow it" is not enough, given that
> there is a substantial cost to disallowing it.
>
> If you don't think there's a cost, consider that we *both*
> seem to be having trouble predicting the consequences of
> your proposed syntax, and you're the one who invented it.
> That's not a good sign!

FWIW, now that I've seen the precedence table in the updated PEP, it
seems really natural to me:
   https://www.python.org/dev/peps/pep-0492/#updated-operator-precedence-table
According to that, "await" is just a prefix operator that binds more
tightly than any arithmetic operation, but less tightly than
indexing/funcall/attribute lookup, which seems about right.

However, if what I just wrote were true, then that would mean that
"await -foo" and "await await foo" would be syntactically legal
(though useless). The fact that they apparently are *not* legal means
that in fact there is still some weird thing going on in the syntax
that I don't understand. And the PEP gives no further details, it just
suggests I go read the parser generator source.

My preference would be that the PEP be updated so that my one-sentence
summary above became correct. But like Guido, I don't necessarily care
about the exact details all that much. What I do feel strongly about
is that whatever syntax we end up with, there should be *some*
accurate human-readable description of *what it is*. AFAICT the PEP
currently doesn't have that.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Nathaniel,

On 2015-04-29 7:35 PM, Nathaniel Smith wrote:

What I do feel strongly about
is that whatever syntax we end up with, there should be*some*
accurate human-readable description of*what it is*. AFAICT the PEP
currently doesn't have that.


How to define human-readable description of how unary
minus operator works?

Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Guido van Rossum]

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov 
wrote:

> Nathaniel,
>
> On 2015-04-29 7:35 PM, Nathaniel Smith wrote:
>
>> What I do feel strongly about
>> is that whatever syntax we end up with, there should be*some*
>> accurate human-readable description of*what it is*. AFAICT the PEP
>> currently doesn't have that.
>>
>
> How to define human-readable description of how unary
> minus operator works?
>

In a PEP you should probably give grammar that is not the actual grammar
from the implementation, but matches the grammar used in the reference
manual on docs.python.org.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Nathaniel Smith]

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov  wrote:
> Nathaniel,
>
> On 2015-04-29 7:35 PM, Nathaniel Smith wrote:
>>
>> What I do feel strongly about
>> is that whatever syntax we end up with, there should be*some*
>> accurate human-readable description of*what it is*. AFAICT the PEP
>> currently doesn't have that.
>
> How to define human-readable description of how unary
> minus operator works?

Hah, good question :-). Of course we all learned how to parse
arithmetic in school, so perhaps it's a bit cheating to refer to that
knowledge. Except of course basically all our users *do* have that
knowledge (or else are forced to figure it out anyway). So I would be
happy with a description of "await" that just says "it's like unary
minus but higher precedence".

Even if we put aside our trained intuitions about arithmetic, I think
it's correct to say that the way unary minus is parsed is: everything
to the right of it that has a tighter precedence gets collected up and
parsed as an expression, and then it takes that expression as its
argument. Still pretty simple.

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Guido,

On 2015-04-29 7:52 PM, Guido van Rossum wrote:

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov 
wrote:


Nathaniel,

On 2015-04-29 7:35 PM, Nathaniel Smith wrote:


What I do feel strongly about
is that whatever syntax we end up with, there should be*some*
accurate human-readable description of*what it is*. AFAICT the PEP
currently doesn't have that.


How to define human-readable description of how unary
minus operator works?


In a PEP you should probably give grammar that is not the actual grammar
from the implementation, but matches the grammar used in the reference
manual on docs.python.org.


Will do!

Thanks,
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Yury Selivanov]

Nathaniel,

On 2015-04-29 7:58 PM, Nathaniel Smith wrote:

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov  wrote:

Nathaniel,

On 2015-04-29 7:35 PM, Nathaniel Smith wrote:

What I do feel strongly about
is that whatever syntax we end up with, there should be*some*
accurate human-readable description of*what it is*. AFAICT the PEP
currently doesn't have that.

How to define human-readable description of how unary
minus operator works?

Hah, good question :-). Of course we all learned how to parse
arithmetic in school, so perhaps it's a bit cheating to refer to that
knowledge. Except of course basically all our users *do* have that
knowledge (or else are forced to figure it out anyway). So I would be
happy with a description of "await" that just says "it's like unary
minus but higher precedence".

Even if we put aside our trained intuitions about arithmetic, I think
it's correct to say that the way unary minus is parsed is: everything
to the right of it that has a tighter precedence gets collected up and
parsed as an expression, and then it takes that expression as its
argument. Still pretty simple.




Well, await is defined exactly like that ;)

Anyways, I'll follow Guido's suggestion to define
await in the PEP the same way we define other syntax
in python docs.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Nathaniel Smith]

On Wed, Apr 29, 2015 at 5:05 PM, Yury Selivanov  wrote:
> Nathaniel,
>
> On 2015-04-29 7:58 PM, Nathaniel Smith wrote:
>>
>> On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov 
>> wrote:
>>>
>>> Nathaniel,
>>>
>>> On 2015-04-29 7:35 PM, Nathaniel Smith wrote:

 What I do feel strongly about
 is that whatever syntax we end up with, there should be*some*
 accurate human-readable description of*what it is*. AFAICT the PEP
 currently doesn't have that.
>>>
>>> How to define human-readable description of how unary
>>> minus operator works?
>>
>> Hah, good question :-). Of course we all learned how to parse
>> arithmetic in school, so perhaps it's a bit cheating to refer to that
>> knowledge. Except of course basically all our users *do* have that
>> knowledge (or else are forced to figure it out anyway). So I would be
>> happy with a description of "await" that just says "it's like unary
>> minus but higher precedence".
>>
>> Even if we put aside our trained intuitions about arithmetic, I think
>> it's correct to say that the way unary minus is parsed is: everything
>> to the right of it that has a tighter precedence gets collected up and
>> parsed as an expression, and then it takes that expression as its
>> argument. Still pretty simple.
>
>
> Well, await is defined exactly like that ;)

So you're saying that "await -fut" and "await await fut" are actually
legal syntax after all, contra what the PEP says? Because "- -fut" is
totally legal syntax, so if await and unary minus work the same...
(Again I don't care about those examples in their own right, I just
find it frustrating that I can't answer these questions without asking
you each time.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492 quibble and request

[Ethan Furman]

>From the PEP:

> Why not a __future__ import
>
> __future__ imports are inconvenient and easy to forget to add.

That is a horrible rationale for not using an import.  By that logic we
should have everything in built-ins.  ;)


> Working example
> ...

The working example only uses async def and await, not async with
nor async for nor __aenter__, etc., etc.

Could you put in a more complete example -- maybe a basic chat room
with both server and client -- that demonstrated more of the new
possibilities?

Having gone through the PEP again, I am still no closer to understanding
what happens here:

  data = await reader.read(8192)

What does the flow of control look like at the interpreter level?

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Clarification of PEP 476 "opting out" section

[Nick Coghlan]

Hi folks,

This is just a note to highlight the fact that I tweaked the "Opting
out" section in PEP 476 based on various discussions I've had over the
past few months: https://hg.python.org/peps/rev/dfd96ee9d6a8

The notable changes:

* the example monkeypatching code handles AttributeError when looking
up "ssl._create_unverified_context", in order to accommodate older
versions of Python that don't have PEP 476 implemented
* new paragraph making it clearer that while the intended use case for
the monkeypatching trick is as a workaround to handle environments
where you *know* HTTPS certificate verification won't work properly
(including explicit references to sitecustomize.py and Standard
Operating Environments for Python), there's also a secondary use case
in allowing applications to provide a system administrator controlled
setting to globally disable certificate verification (hence the change
to the example code)
* new paragraph making it explicit that even though we've improved
Python's default behaviour, particularly security sensitive
applications should still provide their own context rather than
relying on the defaults

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Yury Selivanov]

Hi Ethan,

On 2015-04-29 8:21 PM, Ethan Furman wrote:

 From the PEP:


Why not a __future__ import

__future__ imports are inconvenient and easy to forget to add.

That is a horrible rationale for not using an import.  By that logic we
should have everything in built-ins.  ;)



Working example
...

The working example only uses async def and await, not async with
nor async for nor __aenter__, etc., etc.

Could you put in a more complete example -- maybe a basic chat room
with both server and client -- that demonstrated more of the new
possibilities?


Andrew Svetlov has implemented some new features in his
aiomysql driver:

https://github.com/aio-libs/aiomysql/blob/await/tests/test_async_iter.py

I don't want to cite it in the PEP because it's not complete
yet, and some idioms (like 'async with') aren't used to their
full potential.



Having gone through the PEP again, I am still no closer to understanding
what happens here:

   data = await reader.read(8192)

What does the flow of control look like at the interpreter level?


'await' is semantically equivalent to 'yield from' in this line.

To really understand all implementation details of this line
you need to read PEP 3156 and experiment with asyncio. There
is no easier way, unfortunately.  I can't add a super detailed
explanation how event loops can be implemented in PEP 492,
that's not in its scope.

The good news is that to use asyncio on a daily basis you
don't need to know all details, as you don't need to know
how 'ceval.c' works and how 'setTimeout' is implemented in
JavaScript.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Nick Coghlan]

On 30 April 2015 at 10:21, Ethan Furman  wrote:
> From the PEP:
>
>> Why not a __future__ import
>>
>> __future__ imports are inconvenient and easy to forget to add.
>
> That is a horrible rationale for not using an import.  By that logic we
> should have everything in built-ins.  ;)

It is also makes things more painful than they need to be for syntax
highlighters. 'as' went through the "not really a keyword" path, and
it's a recipe for complexity in the code generation toolchain and
general quirkiness as things behave in unexpected ways.

We have a defined process for introducing new keywords (i.e.
__future__ imports) and the PEP doesn't adequately make the case for
why we shouldn't use it here.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Stephen J. Turnbull]

Adam Bartoš writes:

 > I am in Windows and my terminal isn't utf-8 at the beginning, but I
 > install custom sys.std* objects at runtime and I also install
 > custom readline hook,

IIRC, on the Linux console and in an uxterm, PYTHONIOENCODING=utf-8 in
the environment does what you want.  (Can't test at the moment, I'm on
a Mac and Terminal.app somehow fails to pass the right thing to Python
from the input methods I have available -- I get an empty string,
while I don't seem to have an uxterm, only an xterm.)  This has to be
set at interpreter startup; once the interpreter has decided its IO
encoding, you can't change it, you can only override it by
intercepting the console input and decoding it yourself.

Regarding your environment, the repeated use of "custom" is a red
flag.  Unless you bundle your whole environment with the code you
distribute, Python can know nothing about that.  In general, Python
doesn't know what encoding it is receiving text in.

If you *do* know, you can set PyCF_SOURCE_IS_UTF8.  So if you know
that all of your users will have your custom stdio and readline hooks
installed (AFAICS, they can't use IDLE or IPython!), then you can
bundle Python built with the flag set, or perhaps you can do the
decoding in your custom stdio module.

Note that even if you have a UTF-8 input source, some users are likely
to be surprised because IIRC Python doesn't canonicalize in its
codecs; that is left for higher-level libraries.  Linux UTF-8 is
usually NFC normalized, while Mac UTF-8 is NFD normalized.

 > >> u'\xce\xb1'

Note that that is perfectly legal Unicode.

 > >>> Le 29 avr. 2015 10:36, "Adam Bartoš"  a écrit :
 > >>> > Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
 > >>> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.

Just to be clear, you accept those results as correct, right?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Guido van Rossum]

On Wed, Apr 29, 2015 at 5:59 PM, Nick Coghlan  wrote:

> On 30 April 2015 at 10:21, Ethan Furman  wrote:
> > From the PEP:
> >
> >> Why not a __future__ import
> >>
> >> __future__ imports are inconvenient and easy to forget to add.
> >
> > That is a horrible rationale for not using an import.  By that logic we
> > should have everything in built-ins.  ;)
>

This response is silly. The point is not against import but against
__future__. A __future__ import definitely is inconvenient -- few people I
know could even recite the correct constraints on their placement.


> It is also makes things more painful than they need to be for syntax
> highlighters.


Does it? Do highlighters even understand __future__ imports? I wouldn't
mind if a highlighter always highlighted 'async' and 'await' as keywords
even where they aren't yet -- since they will be in 3.7.


> 'as' went through the "not really a keyword" path, and
> it's a recipe for complexity in the code generation toolchain and
> general quirkiness as things behave in unexpected ways.
>

I don't recall that -- but it was a really long time ago so I may
misremember (did we even have __future__ at the time?).


> We have a defined process for introducing new keywords (i.e.
> __future__ imports) and the PEP doesn't adequately make the case for
> why we shouldn't use it here.
>

That's fair. But because of the difficulty in introducing new keywords,
many proposals have been shot down or discouraged (or changed to use
punctuation characters or abuse existing keywords) -- we should give Yury
some credit for figuring out a way around this. Honestly I'm personally on
the fence.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

Hi python-dev,

New version of the PEP is attached. Summary of updates:


1. Terminology:

- *native coroutine* term is used for "async def" functions.

- *generator-based coroutine* term is used for PEP 380
coroutines used in asyncio.

- *coroutine* is used when *native coroutine* or
*generator based coroutine* can be used in the same
context.

I think that it's not really productive to discuss the
terminology that we use in the PEP.  Its only purpose is
to disambiguate concepts used in the PEP.  We should discuss
how we will name new 'async def' coroutines in Python
Documentation if the PEP is accepted.  Although if you
notice that somewhere in the PEP it is not crystal clear
what "coroutine" means please give me a heads up!


2. Syntax of await expressions is now thoroghly defined
in the PEP. See "Await Expression", "Updated operator
precedence table", and "Examples of "await" expressions"
sections.

I like the current approach.  I'm still not convinced
that we should make 'await' the same grammatically as
unary minus.

I don't understand why we should allow parsing things
like 'await -fut'; this should be a SyntaxError.
I'm fine to modify the grammar to allow 'await await fut'
syntax, though.  And I'm open to discussion :)


3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.


4. asyncio / Migration strategy.  Existing code can
be used with PEP 492 as is, everything will work as
expected.

However, since *plain generator*s (not decorated with
@asyncio.coroutine) cannot 'yield from' native coroutines
(async def functions), it might break some code
*while adapting it to the new syntax*.

I'm open to just throw a RuntimeWarning in this case
in 3.5, and raise a TypeError in 3.6.

Please see the "Differences from generators" section of
the PEP.


5. inspect.isawaitable() function. And, all new functions
are now listed in the "New Standard Library Functions"
section.


6. Lot's of small updates and tweaks throughout the PEP.


Thanks,
Yury



PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015


Abstract


This PEP introduces new syntax for coroutines, asynchronous ``with``
statements and ``for`` loops.  The main motivation behind this proposal
is to streamline writing and maintaining asynchronous code, as well as
to simplify previously hard to implement code patterns.


Rationale and Goals
===

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:

* it is easy to confuse coroutines with regular generators, since they
  share the same syntax; async libraries often attempt to alleviate
  this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);

* it is not possible to natively define a coroutine which has no
  ``yield`` or  ``yield from`` statements, again requiring the use of
  decorators to fix potential refactoring issues;

* support for asynchronous calls is limited to expressions where
  ``yield`` is allowed syntactically, limiting the usefulness of
  syntactic features, such as ``with`` and ``for`` statements.

This proposal makes coroutines a native Python language feature, and
clearly separates them from generators.  This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library.  This also enables
linters and IDEs to improve static code analysis and refactoring.

Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.


Specification
=

This proposal introduces new syntax and semantics to enhance coroutine
support in Python.

This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380).  Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).

From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax.  *coroutine* is used in contexts where both
definitions are applicable.


New Coroutine Declaration Syntax


The following ne

Re: [Python-Dev] Unicode literals in Python 2.7

[Chris Angelico]

On Thu, Apr 30, 2015 at 11:03 AM, Stephen J. Turnbull
 wrote:
> Note that even if you have a UTF-8 input source, some users are likely
> to be surprised because IIRC Python doesn't canonicalize in its
> codecs; that is left for higher-level libraries.  Linux UTF-8 is
> usually NFC normalized, while Mac UTF-8 is NFD normalized.
>
>  > >> u'\xce\xb1'
>
> Note that that is perfectly legal Unicode.

It's legal Unicode, but it doesn't mean what he typed in. This means:

'\xce' LATIN CAPITAL LETTER I WITH CIRCUMFLEX
'\xb1' PLUS-MINUS SIGN

but the original input was:

'\u03b1' GREEK SMALL LETTER ALPHA

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Nick Coghlan]

On 30 April 2015 at 10:33, Yury Selivanov  wrote:
> To really understand all implementation details of this line
> you need to read PEP 3156 and experiment with asyncio. There
> is no easier way, unfortunately.  I can't add a super detailed
> explanation how event loops can be implemented in PEP 492,
> that's not in its scope.

This request isn't about understanding the implementation details,
it's about understanding what Python *users* will gain from the PEP
without *needing* to understand the implementation details.

For that, I'd like to see some not-completely-trivial example code in
(or at least linked from) the PEP written using:

* trollius (no "yield from", Python 2 compatible, akin to Twisted's
inlineDeferred's)
* asyncio/tulip ("yield from", Python 3.3+ compatible)
* PEP 492 (async/await, Python 3.5+ only)

The *intent* of PEP 492, like PEP 380 and PEP 342 before it, is "make
asynchronous programming in Python easier". I think it will actually
succeed in that goal, but I don't think it currently does an
especially job of explaining that to folks that aren't both already
deeply invested in the explicitly asynchronous programming model *and*
thoroughly aware of the fact that most of us need asynchronous
programming to look as much like synchronous programming as possible
in order for it to fit our brains. Some folks can fit ravioli code
with callbacks going everywhere in their brains, but I can't, and it's
my experience that most other folks can't either. This lack means the
PEP that gets confused objections from folks that wish explicitly
asynchronous programming models would just go away entirely (they
won't), as well as from folks that already understand it and don't see
why we can't continue treating it as a special case of other features
that folks have to learn how to use from first principles, rather than
saving them that up front learning cost by doing the pattern
extraction to make event driven explicitly asynchronous programming
its own first class concept with dedicated syntax (a hint on that
front: with statements are just particular patterns for using
try/except/finally, decorators are just particular patterns in using
higher order functions, for statements are just particular patterns in
using while statements and builtins, and even imports and classes just
represent particular patterns in combining dictionaries, code
execution and the metaclass machinery - the pattern extraction and
dedicated syntax associated with all of them makes it possible to
learn to *use* these concepts without first having to learn how to
*implement* them)

>From my own perspective, I've spent a reasonable amount of time
attempting to explain to folks the "modal" nature of generators, in
that you can use them both as pure iterable data sources *and* as
coroutines (as per PEP 342).

The problem I've found is that our current approach ends up failing
the "conceptually different things should also look superficially
different" test: outside certain data pipeline processing problems,
generators-as-iterators and generators-as-coroutines mostly end up
being fundamentally *different* ways of approaching a programming
problem, but the current shared syntax encourages users to attempt to
place them in the same mental bucket. Those users will remain
eternally confused until they learn to place them in two different
buckets despite the shared syntax (the 5 or so different meanings of
"static" in C++ come to mind at this point...).

With explicitly asynchronous development*, this problem of needing to
learn to segment the world into two pieces is particularly important,
and this wonderful rant on red & blue functions published a couple of
months ago helps explain why:
http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

The dedicated async/await syntax proposed in PEP 492 lets the end user
mostly *not care* about all the messiness that needs to happen under
the covers to make explicitly asynchronous programming as close to
normal synchronous programming as possible. The fact that under the
hood in CPython normal functions, coroutines, and generators are
implemented largely as variant behaviours of the same type (producing
respectively the requested result, an object expecting to be driven by
an event loop to produce the requested result, and an object expecting
to be driven be an iterative loop to produce a succession of requested
values rather than a single result) can (and at least arguably should)
be irrelevant to the mental model formed by future Python programmers
(see http://uxoslo.com/2014/01/14/ux-hints-why-mental-models-matter/
for more on the difference between the representational models we
present directly to end users and the underlying implementation models
we use as programmers to actually make things work)

Regards,
Nick.

P.S. *While it's not a substitute for explicitly asynchronous
development, implicitly asynchronous code still has an important role
to play as one of the 3 models

Re: [Python-Dev] PEP 492 quibble and request

[Nick Coghlan]

On 30 April 2015 at 11:12, Guido van Rossum  wrote:
> On Wed, Apr 29, 2015 at 5:59 PM, Nick Coghlan  wrote:
>> It is also makes things more painful than they need to be for syntax
>> highlighters.
>
> Does it? Do highlighters even understand __future__ imports? I wouldn't mind
> if a highlighter always highlighted 'async' and 'await' as keywords even
> where they aren't yet -- since they will be in 3.7.

Yeah, that's a good point.

>> 'as' went through the "not really a keyword" path, and
>> it's a recipe for complexity in the code generation toolchain and
>> general quirkiness as things behave in unexpected ways.
>
> I don't recall that -- but it was a really long time ago so I may
> misremember (did we even have __future__ at the time?).

I don't actually know, I only know about its former pseudo-keyword
status because we made it a real keyword as part of "from __future__
import with_statement".

I think I was conflating it with the hassles we encountered at various
points due to None, True, and False not being real keywords in Python
2, but I don't believe the problems we had with those apply here
(given that we won't be using 'await' and 'async' as values in any
context that the bytecode generation chain cares about).

>> We have a defined process for introducing new keywords (i.e.
>> __future__ imports) and the PEP doesn't adequately make the case for
>> why we shouldn't use it here.
>
> That's fair. But because of the difficulty in introducing new keywords, many
> proposals have been shot down or discouraged (or changed to use punctuation
> characters or abuse existing keywords) -- we should give Yury some credit
> for figuring out a way around this. Honestly I'm personally on the fence.

Yeah, I'm coming around to the idea. For the async pseudo-keyword, I
can see that the proposal only allows its use in cases that were
previously entirely illegal, but I'm not yet clear on how the PEP
proposes to avoid changing the meaning of the following code:

x = await(this_is_a_function_call)

Unless I'm misreading the proposed grammar in the PEP (which is
entirely possible), I believe PEP 492 would reinterpret that as:

x = await this_is_not_a_function_call_any_more

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Yury Selivanov]

One more thing to discuss:

7. StopAsyncIteration vs AsyncStopIteration.

I don't have a strong opinion on this, I prefer
the former because it reads better.

There was no consensus on which one we should use.

Thanks,

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Yury Selivanov]

Nick,

On 2015-04-29 9:52 PM, Nick Coghlan wrote:

On 30 April 2015 at 10:33, Yury Selivanov  wrote:

To really understand all implementation details of this line
you need to read PEP 3156 and experiment with asyncio. There
is no easier way, unfortunately.  I can't add a super detailed
explanation how event loops can be implemented in PEP 492,
that's not in its scope.

This request isn't about understanding the implementation details,
it's about understanding what Python *users* will gain from the PEP
without *needing* to understand the implementation details.

For that, I'd like to see some not-completely-trivial example code in
(or at least linked from) the PEP written using:

* trollius (no "yield from", Python 2 compatible, akin to Twisted's
inlineDeferred's)
* asyncio/tulip ("yield from", Python 3.3+ compatible)
* PEP 492 (async/await, Python 3.5+ only)


I'll see what I can do with aiomysql library to showcase
async for and async with.  I have a few outstanding tasks
with reference implementation to test/add though.

I'm not sure that trollius will add anything to it.  It's just
like asyncio, but uses 'yield' instead of 'yield from' and was
envisioned by Victor Stinner as a transitional-framework
to ease the porting of openstack to python 3.



The *intent* of PEP 492, like PEP 380 and PEP 342 before it, is "make
asynchronous programming in Python easier". I think it will actually
succeed in that goal, but I don't think it currently does an
especially job of explaining that to folks that aren't both already
deeply invested in the explicitly asynchronous programming model *and*
thoroughly aware of the fact that most of us need asynchronous
programming to look as much like synchronous programming as possible
in order for it to fit our brains. Some folks can fit ravioli code
with callbacks going everywhere in their brains, but I can't, and it's
my experience that most other folks can't either. This lack means the
PEP that gets confused objections from folks that wish explicitly
asynchronous programming models would just go away entirely (they
won't), as well as from folks that already understand it and don't see
why we can't continue treating it as a special case of other features
that folks have to learn how to use from first principles, rather than
saving them that up front learning cost by doing the pattern
extraction to make event driven explicitly asynchronous programming
its own first class concept with dedicated syntax (a hint on that
front: with statements are just particular patterns for using
try/except/finally, decorators are just particular patterns in using
higher order functions, for statements are just particular patterns in
using while statements and builtins, and even imports and classes just
represent particular patterns in combining dictionaries, code
execution and the metaclass machinery - the pattern extraction and
dedicated syntax associated with all of them makes it possible to
learn to *use* these concepts without first having to learn how to
*implement* them)

 From my own perspective, I've spent a reasonable amount of time
attempting to explain to folks the "modal" nature of generators, in
that you can use them both as pure iterable data sources *and* as
coroutines (as per PEP 342).

The problem I've found is that our current approach ends up failing
the "conceptually different things should also look superficially
different" test: outside certain data pipeline processing problems,
generators-as-iterators and generators-as-coroutines mostly end up
being fundamentally *different* ways of approaching a programming
problem, but the current shared syntax encourages users to attempt to
place them in the same mental bucket. Those users will remain
eternally confused until they learn to place them in two different
buckets despite the shared syntax (the 5 or so different meanings of
"static" in C++ come to mind at this point...).


Agree.  This confusion of trying to fit two fundamentally
different programming models in one syntax is what led me
to start thinking about PEP 492 ideas several years ago.

And the absence of 'async with' and 'async for' statements
forced me to use greenlets, which is another reason for
the PEP.



With explicitly asynchronous development*, this problem of needing to
learn to segment the world into two pieces is particularly important,
and this wonderful rant on red & blue functions published a couple of
months ago helps explain why:
http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

The dedicated async/await syntax proposed in PEP 492 lets the end user
mostly *not care* about all the messiness that needs to happen under
the covers to make explicitly asynchronous programming as close to
normal synchronous programming as possible. The fact that under the
hood in CPython normal functions, coroutines, and generators are
implemented largely as variant behaviours of the same type (producing
respectively the requested result, an object ex

Re: [Python-Dev] PEP 492 quibble and request

[Guido van Rossum]

On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan  wrote:

> [...]
> Yeah, I'm coming around to the idea. For the async pseudo-keyword, I
> can see that the proposal only allows its use in cases that were
> previously entirely illegal, but I'm not yet clear on how the PEP
> proposes to avoid changing the meaning of the following code:
>
> x = await(this_is_a_function_call)
>
> Unless I'm misreading the proposed grammar in the PEP (which is
> entirely possible), I believe PEP 492 would reinterpret that as:
>
> x = await this_is_not_a_function_call_any_more
>

Ah, but here's the other clever bit: it's only interpreted this way
*inside* a function declared with 'async def'. Outside such functions,
'await' is not a keyword, so that grammar rule doesn't trigger. (Kind of
similar to the way that the print_function __future__ disables the
keyword-ness of 'print', except here it's toggled on or off depending on
whether the nearest surrounding scope is 'async def' or not. The PEP could
probably be clearer about this; it's all hidden in the Transition Plan
section.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Nick Coghlan]

On 30 April 2015 at 12:31, Guido van Rossum  wrote:
> On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan  wrote:
>>
>> [...]
>> Yeah, I'm coming around to the idea. For the async pseudo-keyword, I
>> can see that the proposal only allows its use in cases that were
>> previously entirely illegal, but I'm not yet clear on how the PEP
>> proposes to avoid changing the meaning of the following code:
>>
>> x = await(this_is_a_function_call)
>>
>> Unless I'm misreading the proposed grammar in the PEP (which is
>> entirely possible), I believe PEP 492 would reinterpret that as:
>>
>> x = await this_is_not_a_function_call_any_more
>
>
> Ah, but here's the other clever bit: it's only interpreted this way *inside*
> a function declared with 'async def'. Outside such functions, 'await' is not
> a keyword, so that grammar rule doesn't trigger. (Kind of similar to the way
> that the print_function __future__ disables the keyword-ness of 'print',
> except here it's toggled on or off depending on whether the nearest
> surrounding scope is 'async def' or not. The PEP could probably be clearer
> about this; it's all hidden in the Transition Plan section.)

Ah, nice, although even reading the Transition Plan section didn't
clue me in to that particular aspect of the idea :)

Given that clarification, I think the rationale for "no __future__
statement needed" can be strengthened by focusing on the fact that
such a statement would largely be *redundant*, given that:

* "async def", "async with", and "async for" are all currently syntax
errors, and hence adding them is backwards compatible if "async" is
otherwise treated as a normal variable name
* "await " only gains its new interpretation when used inside an
"async def" statement, so "async def" fills the role that a module
level compiler declaration like "from __future__ import
async_functions" would otherwise fill

That said, it may be worth having the future statement *anyway* as:

1. It gives us the runtime accessible record of the feature transition
in the __future__ module
2. It lets folks opt-in to the full keyword implementation from day 1
if they prefer, addressing the tokenisation limitation noted in the
PEP: https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Yury Selivanov]

On 2015-04-29 11:01 PM, Nick Coghlan wrote:

On 30 April 2015 at 12:31, Guido van Rossum  wrote:

>On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan  wrote:

>>
>>[...]
>>Yeah, I'm coming around to the idea. For the async pseudo-keyword, I
>>can see that the proposal only allows its use in cases that were
>>previously entirely illegal, but I'm not yet clear on how the PEP
>>proposes to avoid changing the meaning of the following code:
>>
>> x = await(this_is_a_function_call)
>>
>>Unless I'm misreading the proposed grammar in the PEP (which is
>>entirely possible), I believe PEP 492 would reinterpret that as:
>>
>> x = await this_is_not_a_function_call_any_more

>
>
>Ah, but here's the other clever bit: it's only interpreted this way*inside*
>a function declared with 'async def'. Outside such functions, 'await' is not
>a keyword, so that grammar rule doesn't trigger. (Kind of similar to the way
>that the print_function __future__ disables the keyword-ness of 'print',
>except here it's toggled on or off depending on whether the nearest
>surrounding scope is 'async def' or not. The PEP could probably be clearer
>about this; it's all hidden in the Transition Plan section.)

Ah, nice, although even reading the Transition Plan section didn't
clue me in to that particular aspect of the idea :)

Given that clarification, I think the rationale for "no __future__
statement needed" can be strengthened by focusing on the fact that
such a statement would largely be*redundant*, given that:

* "async def", "async with", and "async for" are all currently syntax
errors, and hence adding them is backwards compatible if "async" is
otherwise treated as a normal variable name
* "await " only gains its new interpretation when used inside an
"async def" statement, so "async def" fills the role that a module
level compiler declaration like "from __future__ import
async_functions" would otherwise fill


Thanks, Nick.

I've fixed the Transition Plan section, and rewrote the
"why not __future__" one too.

https://hg.python.org/peps/rev/552773d7e085
https://hg.python.org/peps/rev/5db3ad3d540b

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Guido van Rossum]

Belt and suspenders. :-)

We may need help with the implementation though -- PEP 479 is still waiting
on implementation help with __future__ too AFAIK.

On Wed, Apr 29, 2015 at 8:01 PM, Nick Coghlan  wrote:

> On 30 April 2015 at 12:31, Guido van Rossum  wrote:
> > On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan 
> wrote:
> >>
> >> [...]
> >> Yeah, I'm coming around to the idea. For the async pseudo-keyword, I
> >> can see that the proposal only allows its use in cases that were
> >> previously entirely illegal, but I'm not yet clear on how the PEP
> >> proposes to avoid changing the meaning of the following code:
> >>
> >> x = await(this_is_a_function_call)
> >>
> >> Unless I'm misreading the proposed grammar in the PEP (which is
> >> entirely possible), I believe PEP 492 would reinterpret that as:
> >>
> >> x = await this_is_not_a_function_call_any_more
> >
> >
> > Ah, but here's the other clever bit: it's only interpreted this way
> *inside*
> > a function declared with 'async def'. Outside such functions, 'await' is
> not
> > a keyword, so that grammar rule doesn't trigger. (Kind of similar to the
> way
> > that the print_function __future__ disables the keyword-ness of 'print',
> > except here it's toggled on or off depending on whether the nearest
> > surrounding scope is 'async def' or not. The PEP could probably be
> clearer
> > about this; it's all hidden in the Transition Plan section.)
>
> Ah, nice, although even reading the Transition Plan section didn't
> clue me in to that particular aspect of the idea :)
>
> Given that clarification, I think the rationale for "no __future__
> statement needed" can be strengthened by focusing on the fact that
> such a statement would largely be *redundant*, given that:
>
> * "async def", "async with", and "async for" are all currently syntax
> errors, and hence adding them is backwards compatible if "async" is
> otherwise treated as a normal variable name
> * "await " only gains its new interpretation when used inside an
> "async def" statement, so "async def" fills the role that a module
> level compiler declaration like "from __future__ import
> async_functions" would otherwise fill
>
> That said, it may be worth having the future statement *anyway* as:
>
> 1. It gives us the runtime accessible record of the feature transition
> in the __future__ module
> 2. It lets folks opt-in to the full keyword implementation from day 1
> if they prefer, addressing the tokenisation limitation noted in the
> PEP:
> https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Greg Ewing]

Paul Moore wrote:

I agree. While I don't use coroutines/asyncio, and I may never do so,
I will say that I find Python's approach very difficult to understand.


Well, I tried to offer something easier to understand.
The idea behind PEP 3152 is that writing async code
should be just like writing threaded code, except that
the suspension points are explicit. But apparently
that was too simple, or something.


Looking at the Wikipedia article on coroutines, I see an example of
how a producer/consumer process might be written with coroutines:

var q := new queue

coroutine produce
loop
while q is not full
create some new items
add the items to q
yield to consume

coroutine consume
loop
while q is not empty
remove some items from q
use the items
yield to produce


Aaargh, this is what we get for overloading the word
"coroutine". The Wikipedia article is talking about a
technique where coroutines yield control to other
explicitly identified coroutines.

Coroutines in asyncio don't work that way; instead
they just suspend themselves, and the event loop
takes care of deciding which one to run next.


I can't even see how to relate that to PEP 429 syntax. I'm not allowed
to use "yield",


You probably wouldn't need to explicitly yield, since
you'd use an asyncio.Queue for passing data between the
tasks, which takes care of suspending until data
becomes available.

You would only need to yield if you were implementing
some new synchronisation primitive. Yury's answer to
that appears to be that you don't do it with an async
def function, you create an object that implements
the awaitable-object protocol directly.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 vs. PEP 3152, new round

[Greg Ewing]

Nathaniel Smith wrote:
(I suspect this may also be the impetus behind Greg's request that it 
just be treated the same as unary minus. IMHO it matters much more that 
the rules be predictable and teachable than that they allow or disallow 
every weird edge case in exactly the right way.)


Exactly.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Greg Ewing]

Yury Selivanov wrote:

Everybody is pulling me in a different direction :)
Guido proposed to call them "native coroutines".  Some people
think that "async functions" is a better name.  Greg loves
his "cofunction" term.


Not the term, the *idea*. But PEP 492 is not based on
that idea, so I don't advocate calling anything in
PEP 492 a cofunction. My vote goes to "async function".

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: What is the real goal?

[Greg Ewing]

Skip Montanaro wrote:
According to Wikipedia , term 
"coroutine" was first coined in 1958, so several generations of computer 
science graduates will be familiar with the textbook definition. If your 
use of "coroutine" matches the textbook definition of the term, I think 
you should continue to use it instead of inventing new names which will 
just confuse people new to Python.


I don't think anything in asyncio or PEP 492 fits that
definition directly. Generators and async def functions
seem to be what that page calls a "generator" or "semicoroutine":

   they differ in that coroutines can control where execution
   continues after they yield, while generators cannot, instead
   transferring control back to the generator's caller.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com