Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Eric V. Smith 

On 11/3/2016 3:06 PM, Fabio Zadrozny wrote:

Hi Python-Dev,

I'm trying to get my head around on what's accepted in f-strings --
https://www.python.org/dev/peps/pep-0498/ seems very light on the
details on what it does accept as an expression and how things should
actually be parsed (and the current implementation still doesn't seem to
be in a state for a final release, so, I thought asking on python-dev
would be a reasonable option).


In what way do you think the implementation isn't ready for a final release?


I was thinking there'd be some grammar for it (something as
https://docs.python.org/3.6/reference/grammar.html), but all I could
find related to this is a quote saying that f-strings should be
something as:

f '  {} 

So, given that, is it safe to assume that  would be equal to
the "test" node from the official grammar?


No. There are really three phases here:

1. The f-string is tokenized as a regular STRING token, like all other 
strings (f-, b-, u-, r-, etc).
2. The parser sees that it's an f-string, and breaks it into expression 
and text parts.
3. For each expression found, the expression is compiled with 
PyParser_ASTFromString(..., Py_eval_input, ...).


Step 2 is the part that limits what types of expressions are allowed. 
While scanning for the end of an expression, it stops at the first '!', 
':', or '}' that isn't inside of a string and isn't nested inside of 
parens, braces, and brackets.


The nesting-tracking is why these work:
>>> f'{(lambda x:3)}'
' at 0x0296E560>'
>>> f'{(lambda x:3)!s:.20}'
' a'

But this doesn't:
>>> f'{lambda x:3}'
  File "", line 1
(lambda x)
 ^
SyntaxError: unexpected EOF while parsing

Also, backslashes are not allowed anywhere inside of the expression. 
This was a late change right before beta 1 (I think), and differs from 
the PEP and docs. I have an open item to fix them.



I initially thought it would obviously be, but the PEP says that using a
lamda inside the expression would conflict because of the colon (which
wouldn't happen if a proper grammar was actually used for this parsing
as there'd be no conflict as the lamda would properly consume the
colon), so, I guess some pre-parser steps takes place to separate the
expression to then be parsed, so, I'm interested on knowing how exactly
that should work when the implementation is finished -- lots of plus
points if there's actually a grammar to back it up :)


I've considered using the grammar and tokenizer to implement f-string 
parsing, but I doubt it will ever happen. It's a lot of work, and 
everything that produced or consumed tokens would have to be aware of 
it. As it stands, if you don't need to look inside of f-strings, you can 
just treat them as regular STRING tokens.


I hope that helps.

Eric.

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Simon Cross 
On Fri, Nov 4, 2016 at 9:56 AM, Eric V. Smith  wrote:
> 2. The parser sees that it's an f-string, and breaks it into expression and
> text parts.

I'm with Fabio here. It would be really nice to have a grammar
specified and documented for this step, even if it's not implemented
that way. Otherwise it's going to be very hard for, e.g., syntax
highlighters to know what is intended to be allowed.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Paul Moore 
On 4 November 2016 at 08:36, Simon Cross  wrote:
> On Fri, Nov 4, 2016 at 9:56 AM, Eric V. Smith  wrote:
>> 2. The parser sees that it's an f-string, and breaks it into expression and
>> text parts.
>
> I'm with Fabio here. It would be really nice to have a grammar
> specified and documented for this step, even if it's not implemented
> that way. Otherwise it's going to be very hard for, e.g., syntax
> highlighters to know what is intended to be allowed.

I think that if the docs explain the process, essentially as noted by
Eric above:

> Step 2 is the part that limits what types of expressions are allowed.
> While scanning for the end of an expression, it stops at the first '!', ':', 
> or '}' that
> isn't inside of a string and isn't nested inside of parens, braces, and 
> brackets.
[...]
> Also, backslashes are not allowed anywhere inside of the expression.

then that would be fine. Possibly a bit more detail would be helpful,
but I'm pretty sure I could reimplement the behaviour (for a syntax
highlighter, for example) based on the above.

I assume that the open item Eric mentions to fix the PEP and docs is
sufficient to cover this, so it'll be documented in due course.

Paul
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

2016-11-04 Thread Victor Stinner 
(Re-post without the two attached files of 100 KB each.)

Hi,

You may know that I'm working on benchmarks. I regenerated all
benchmark results of speed.python.org using performance 0.3.2
(benchmark suite). I started to analyze results.

All results are available online on the website:

   https://speed.python.org/


To communicate on my work on benchmarks, I tweeted two pictures:

"sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
2.7 #python #benchmark":
https://twitter.com/VictorStinner/status/794289596683210760

"Python 3.6 is between 25% and 54% slower than Python 2.7 in the
following benchmarks":
https://twitter.com/VictorStinner/status/794305065708376069


Many people were disappointed that Python 3.6 can be up to 54% slower
than Python 2.7. In fact, I know many reasons which explain that, but
it's hard to summarize them in 140 characters ;-)

For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
pycrypto_aes. This benchmark tests a pure Python implementation of the
crypto cipher AES. You may know that CPython is slow for CPU intensive
functions, especially on integer and floatting point numbers.

"int" in Python 3 is now "long integers" by default, which is known to
be a little bit slower than "short int" of Python 2. On a more
realistic benchmark (see other benchmarks), the overhead of Python 3
"long int" is negligible.

AES is a typical example stressing integers. For me, it's a dummy
benchmark: it doesn't make sense to use Python for AES: modern CPUs
have an *hardware* implemention which is super fast.


Well, I didn't have time to analyze in depth individual benchmarks. If
you want to help me, here is the source code of benchmarks:
https://github.com/python/performance/blob/master/performance/benchmarks/


Raw results of Python 3.6 compared to Python 2.7:
---
$ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
Slower (40):
- python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
- python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower
- unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
- call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
- call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
- call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
- crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
- xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
- logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
- logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
- pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
- spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
- logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
- chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
- go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
- xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
- sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
- xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
- django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
- fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
- hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
- chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
- regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
- json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
- nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
- genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
- raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
- scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
- scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
- deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
- sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
- call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
- scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
- meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
- pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
- richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
- genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
- float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower
- scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms:
1.07x slower
- xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower

Faster (15):
- telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster
- unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster
- pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster
- json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster
- pickle: 40.4 us +- 0.6 us -> 27.

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Fabio Zadrozny 
Answers inline...

On Fri, Nov 4, 2016 at 5:56 AM, Eric V. Smith  wrote:

> On 11/3/2016 3:06 PM, Fabio Zadrozny wrote:
>
>> Hi Python-Dev,
>>
>> I'm trying to get my head around on what's accepted in f-strings --
>> https://www.python.org/dev/peps/pep-0498/ seems very light on the
>> details on what it does accept as an expression and how things should
>> actually be parsed (and the current implementation still doesn't seem to
>> be in a state for a final release, so, I thought asking on python-dev
>> would be a reasonable option).
>>
>
> In what way do you think the implementation isn't ready for a final
> release?


Well, the cases listed in the docs​ (https://hg.python.org/
cpython/file/default/Doc/reference/lexical_analysis.rst) don't work in the
latest release (with SyntaxErrors) -- and the bug I created related to it:
http://bugs.python.org/issue28597 was promptly closed as duplicate -- so, I
assumed (maybe wrongly?) that the parsing still needs work.


> I was thinking there'd be some grammar for it (something as
>> https://docs.python.org/3.6/reference/grammar.html), but all I could
>> find related to this is a quote saying that f-strings should be
>> something as:
>>
>> f '  {   > specifier> } 
>>
>> So, given that, is it safe to assume that  would be equal to
>> the "test" node from the official grammar?
>>
>
> No. There are really three phases here:
>
> 1. The f-string is tokenized as a regular STRING token, like all other
> strings (f-, b-, u-, r-, etc).
> 2. The parser sees that it's an f-string, and breaks it into expression
> and text parts.
> 3. For each expression found, the expression is compiled with
> PyParser_ASTFromString(..., Py_eval_input, ...).
>
> Step 2 is the part that limits what types of expressions are allowed.
> While scanning for the end of an expression, it stops at the first '!',
> ':', or '}' that isn't inside of a string and isn't nested inside of
> parens, braces, and brackets.
>

​It'd be nice if at least this description could be added to the PEP (as
all other language implementations and IDEs will have to work the same way
and will probably reference it) -- a grammar example, even if not used
would be helpful (personally, I think hand-crafted parsers are always worse
in the long run compared to having a proper grammar with a parser, although
I understand that if you're not really used to it, it may be more work to
set it up).

Also, I find it a bit troubling that PyParser_ASTFromString is used there
and not just the node which would be related to an expression, although I
understand it's probably an easier approach, although in the end you
probably have to filter it and end up just accepting what's beneath the
"test" from the grammar, no? (i.e.: that's what a lambda body accepts).


> The nesting-tracking is why these work:
> >>> f'{(lambda x:3)}'
> ' at 0x0296E560>'
> >>> f'{(lambda x:3)!s:.20}'
> ' a'
>
> But this doesn't:
> >>> f'{lambda x:3}'
>   File "", line 1
> (lambda x)
>  ^
> SyntaxError: unexpected EOF while parsing
>
> Also, backslashes are not allowed anywhere inside of the expression. This
> was a late change right before beta 1 (I think), and differs from the PEP
> and docs. I have an open item to fix them.
>
> I initially thought it would obviously be, but the PEP says that using a
>> lamda inside the expression would conflict because of the colon (which
>> wouldn't happen if a proper grammar was actually used for this parsing
>> as there'd be no conflict as the lamda would properly consume the
>> colon), so, I guess some pre-parser steps takes place to separate the
>> expression to then be parsed, so, I'm interested on knowing how exactly
>> that should work when the implementation is finished -- lots of plus
>> points if there's actually a grammar to back it up :)
>>
>
> I've considered using the grammar and tokenizer to implement f-string
> parsing, but I doubt it will ever happen. It's a lot of work, and
> everything that produced or consumed tokens would have to be aware of it.
> As it stands, if you don't need to look inside of f-strings, you can just
> treat them as regular STRING tokens.
>

​Well, I think all language implementations / IDEs (or at least those which
want to give syntax errors) will *have* to look inside f-strings.

Also, you could still have a separate grammar saying how to look inside
f-strings (this would make the lives of other implementors easier) even if
it was a post-processing step as you're doing now.


>
> I hope that helps.
>
> Eric.
>

​It does, thank you very much.

​Best Regards,

Fabio​
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

2016-11-04 Thread Victor Stinner 
The nice thing with benchmarks is that you can always take a subset of
numbers to prove something :-) Another subset where Python 3.6 is
faster:

"Python 3.6 is between 12% (1.14x) and 97% (32x) FASTER than Python
2.7 in the following benchmarks:"
https://twitter.com/VictorStinner/status/794525289623719937

Victor

2016-11-04 13:53 GMT+01:00 Victor Stinner :
> (Re-post without the two attached files of 100 KB each.)
>
> Hi,
>
> You may know that I'm working on benchmarks. I regenerated all
> benchmark results of speed.python.org using performance 0.3.2
> (benchmark suite). I started to analyze results.
>
> All results are available online on the website:
>
>https://speed.python.org/
>
>
> To communicate on my work on benchmarks, I tweeted two pictures:
>
> "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
> 2.7 #python #benchmark":
> https://twitter.com/VictorStinner/status/794289596683210760
>
> "Python 3.6 is between 25% and 54% slower than Python 2.7 in the
> following benchmarks":
> https://twitter.com/VictorStinner/status/794305065708376069
>
>
> Many people were disappointed that Python 3.6 can be up to 54% slower
> than Python 2.7. In fact, I know many reasons which explain that, but
> it's hard to summarize them in 140 characters ;-)
>
> For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
> pycrypto_aes. This benchmark tests a pure Python implementation of the
> crypto cipher AES. You may know that CPython is slow for CPU intensive
> functions, especially on integer and floatting point numbers.
>
> "int" in Python 3 is now "long integers" by default, which is known to
> be a little bit slower than "short int" of Python 2. On a more
> realistic benchmark (see other benchmarks), the overhead of Python 3
> "long int" is negligible.
>
> AES is a typical example stressing integers. For me, it's a dummy
> benchmark: it doesn't make sense to use Python for AES: modern CPUs
> have an *hardware* implemention which is super fast.
>
>
> Well, I didn't have time to analyze in depth individual benchmarks. If
> you want to help me, here is the source code of benchmarks:
> https://github.com/python/performance/blob/master/performance/benchmarks/
>
>
> Raw results of Python 3.6 compared to Python 2.7:
> ---
> $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
> 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
> Slower (40):
> - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
> - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x 
> slower
> - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
> - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
> - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
> - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
> - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
> - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
> - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
> - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
> - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
> - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
> - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
> - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
> - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
> - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
> - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
> - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
> - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
> - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
> - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
> - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
> - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
> - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
> - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
> - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
> - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
> - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
> - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
> - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
> - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
> - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
> - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
> - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
> - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
> - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
> - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slo

[Python-Dev] Summary of Python tracker Issues

2016-11-04 Thread Python tracker 

ACTIVITY SUMMARY (2016-10-28 - 2016-11-04)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5547 (+23)
  closed 34821 (+41)
  total  40368 (+64)

Open issues with patches: 2405 


Issues opened (43)
==

#28199: Compact dict resizing is doing too much work
http://bugs.python.org/issue28199  reopened by ned.deily

#28473: mailbox.MH crashes on certain Claws Mail .mh_sequences files
http://bugs.python.org/issue28473  reopened by r.david.murray

#28551: sysconfig.py wrong _PROJECT_BASE for Py2.7 Windows 64bit PC/VS
http://bugs.python.org/issue28551  opened by anselm.kruis

#28552: Distutils fail if sys.executable is None
http://bugs.python.org/issue28552  opened by iamale

#28554: Windows: _socket module fails to compile on "AMD64 Windows7 SP
http://bugs.python.org/issue28554  opened by haypo

#28555: provid also sha-1 and sha-256 also on download links
http://bugs.python.org/issue28555  opened by Big Stone

#28556: typing.py upgrades
http://bugs.python.org/issue28556  opened by gvanrossum

#28557: error message for bad raw readinto
http://bugs.python.org/issue28557  opened by davidszotten

#28559: Unclear error message when raising wrong type of exceptions
http://bugs.python.org/issue28559  opened by Dimitri Merejkowsky

#28562: test_asyncio fails on Android upon calling getaddrinfo()
http://bugs.python.org/issue28562  opened by xdegaye

#28563: Arbitrary code execution in gettext.c2py
http://bugs.python.org/issue28563  opened by Carl Ekerot

#28564: shutil.rmtree is inefficient due to listdir() instead of scand
http://bugs.python.org/issue28564  opened by enkore

#28565: datetime.strptime %Z doesn't produce aware object
http://bugs.python.org/issue28565  opened by lilydjwg

#28567: Bus error / segmentation fault on macOS debug build when using
http://bugs.python.org/issue28567  opened by Alex Croitor

#28569: mock.attach_mock should work with any return value of patch()
http://bugs.python.org/issue28569  opened by anfedorov

#28570: httplib mishandles unknown 1XX status codes
http://bugs.python.org/issue28570  opened by Lukasa

#28571: llist and scipy.stats conflicts, python segfault
http://bugs.python.org/issue28571  opened by Lluís

#28572: IDLE: add tests for config dialog.
http://bugs.python.org/issue28572  opened by terry.reedy

#28573: Python 3.6.0b3 64-bit has no sys._mercurial info
http://bugs.python.org/issue28573  opened by steve.dower

#28574: Update bundled pip
http://bugs.python.org/issue28574  opened by steve.dower

#28576: Uninstalling Py352 x86 with /uninstall option does not remove 
http://bugs.python.org/issue28576  opened by jcrmatos

#28577: ipaddress.ip_network(...).hosts() returns nothing for an IPv4 
http://bugs.python.org/issue28577  opened by era

#28584: ICC compiler check is too permissive
http://bugs.python.org/issue28584  opened by struktured

#28585: Restore docstring of os._isdir
http://bugs.python.org/issue28585  opened by serhiy.storchaka

#28586: Convert os.scandir to Argument Clinic
http://bugs.python.org/issue28586  opened by serhiy.storchaka

#28587: list.index documentation missing start and stop arguments
http://bugs.python.org/issue28587  opened by ChrisRands

#28588: Memory leak in OpenSSL thread state
http://bugs.python.org/issue28588  opened by christian.heimes

#28590: fstring's '{' from escape sequences does not start an expressi
http://bugs.python.org/issue28590  opened by afg984

#28591: imghdr doesn't recognize some jpeg formats
http://bugs.python.org/issue28591  opened by 4simple-org

#28592: Installation freezes on C Runtime Install
http://bugs.python.org/issue28592  opened by Ex Tracheese

#28593: Inconsistent itertools' predicate behaviour
http://bugs.python.org/issue28593  opened by franciscouzo

#28595: shlex.shlex should not augment wordchars
http://bugs.python.org/issue28595  opened by evan_

#28596: on Android _bootlocale on startup relies on too many library m
http://bugs.python.org/issue28596  opened by xdegaye

#28598: RHS not consulted in `str % subclass_of_str` case.
http://bugs.python.org/issue28598  opened by mjpieters

#28601: Ambiguous datetime comparisons should use == rather than 'is' 
http://bugs.python.org/issue28601  opened by p-ganssle

#28602: `tzinfo.fromutc()` fails when used for a fold-aware tzinfo imp
http://bugs.python.org/issue28602  opened by p-ganssle

#28603: traceback module can't format/print unhashable exceptions
http://bugs.python.org/issue28603  opened by Trundle

#28604: Exception raised by python3.5 when using en_GB locale
http://bugs.python.org/issue28604  opened by Guillaume Pasquet (Etenil)

#28607: C implementation of parts of copy.deepcopy
http://bugs.python.org/issue28607  opened by villemoes

#28608: Support creating hardlink using `pathlib`
http://bugs.python.org/issue28608  opened by cool-RR

#28609: argparse claims '*' positional argument is required in error

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Eric V. Smith 

On 11/4/2016 10:50 AM, Fabio Zadrozny wrote:

In what way do you think the implementation isn't ready for a final
release?


Well, the cases listed in the docs​
(https://hg.python.org/cpython/file/default/Doc/reference/lexical_analysis.rst
)
don't work in the latest release (with SyntaxErrors) -- and the bug I
created related to it: http://bugs.python.org/issue28597
 was promptly closed as duplicate
-- so, I assumed (maybe wrongly?) that the parsing still needs work.


It's not the parsing that needs work, it's the documentation. Those 
examples used to work, but the parser was deliberately changed to not 
support them. There's a long discussion on python-ideas about it, 
starting at 
https://mail.python.org/pipermail/python-ideas/2016-August/041727.html



​It'd be nice if at least this description could be added to the PEP (as
all other language implementations and IDEs will have to work the same
way and will probably reference it) -- a grammar example, even if not
used would be helpful (personally, I think hand-crafted parsers are
always worse in the long run compared to having a proper grammar with a
parser, although I understand that if you're not really used to it, it
may be more work to set it up).


I've written a parser generator just to understand how they work, so I'm 
completely sympathetic to this. However, in this case, I don't think it 
would be any easier. I'm basically writing a tokenizer, not an 
expression parser. It's much simpler. The actual parsing is handled by 
PyParser_ASTFromString. And as I state below, you have to also consider 
the parser consumers.



Also, I find it a bit troubling that PyParser_ASTFromString is used
there and not just the node which would be related to an expression,
although I understand it's probably an easier approach, although in the
end you probably have to filter it and end up just accepting what's
beneath the "test" from the grammar, no? (i.e.: that's what a lambda
body accepts).


Using PyParser_ASTFromString is the easiest possible way to do this. 
Given a string, it returns an AST node. What could be simpler?



​Well, I think all language implementations / IDEs (or at least those
which want to give syntax errors) will *have* to look inside f-strings.


While it's probably true that IDEs (and definitely language 
implementations) will want to parse f-strings, I think there are many 
more code scanners that are not language implementations or IDEs. And by 
being "just" regular strings with a new prefix, it's trivial to get any 
parser that doesn't care about the internal structure to at least see 
f-strings as normal strings.



Also, you could still have a separate grammar saying how to look inside
f-strings (this would make the lives of other implementors easier) even
if it was a post-processing step as you're doing now.


Yes. I've contemplated exposing the f-string scanner. That's the part 
that returns expressions (as strings) and literal strings. I realize 
that won't help 3.6.


Eric.

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] itertools predicates

2016-11-04 Thread Raymond Hettinger 

> On Nov 3, 2016, at 3:51 PM, Steven D'Aprano  wrote:
> 
> Before there can be a solution, there first has to be a problem that 
> needs solving. "Lack of consistency" is not necessarily a problem. The 
> intertools functions are quite different, they do different things with 
> different APIs. The question here should not be "why don't these 
> functions take None as an argument?", rather it should be "why should 
> these functions take None as an argument?".

I concur with Steven who articulated the issue perfectly.  There isn't an 
actual problem here that needs to be solved (i.e. not a single user report in 
15 years indicating that the API wasn't meeting the needs for real use-cases).  
I'm disinclined to churn the API unless there is a real need.

FWIW, groupby() has the predicate as an optional argument so that you can write 
groupby('aaabbc') and have it group by value (much like the key-function on 
sorted() is optional).   The two filter variants allow None as the first 
argument only for historical reasons -- once "bool" came along, it would have 
been better to write filter(bool, someiterable) in preference to using None 
which is less clear about its intention.  The takewhile/dropwhile tools didn't 
have the same constraint to match a historical API, so there was an opportunity 
to have a clearer API with a simpler signature.

As Terry suggested, if you have other itertools feature requests, please put 
them on the tracker an assign them to me.

Thank you,


Raymond





___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Fabio Zadrozny 
On Fri, Nov 4, 2016 at 3:15 PM, Eric V. Smith  wrote:

> On 11/4/2016 10:50 AM, Fabio Zadrozny wrote:
>
>> In what way do you think the implementation isn't ready for a final
>> release?
>>
>>
>> Well, the cases listed in the docs​
>> (https://hg.python.org/cpython/file/default/Doc/reference/
>> lexical_analysis.rst
>> > lexical_analysis.rst>)
>> don't work in the latest release (with SyntaxErrors) -- and the bug I
>> created related to it: http://bugs.python.org/issue28597
>>  was promptly closed as duplicate
>> -- so, I assumed (maybe wrongly?) that the parsing still needs work.
>>
>
> It's not the parsing that needs work, it's the documentation. Those
> examples used to work, but the parser was deliberately changed to not
> support them. There's a long discussion on python-ideas about it, starting
> at https://mail.python.org/pipermail/python-ideas/2016-August/041727.html


​Understood ;)
​


>
>
> ​It'd be nice if at least this description could be added to the PEP (as
>> all other language implementations and IDEs will have to work the same
>> way and will probably reference it) -- a grammar example, even if not
>> used would be helpful (personally, I think hand-crafted parsers are
>> always worse in the long run compared to having a proper grammar with a
>> parser, although I understand that if you're not really used to it, it
>> may be more work to set it up).
>>
>
> I've written a parser generator just to understand how they work, so I'm
> completely sympathetic to this. However, in this case, I don't think it
> would be any easier. I'm basically writing a tokenizer, not an expression
> parser. It's much simpler. The actual parsing is handled by
> PyParser_ASTFromString. And as I state below, you have to also consider the
> parser consumers.
>
> Also, I find it a bit troubling that
>> ​​
>> PyParser_ASTFromString is used
>> there and not just the node which would be related to an expression,
>> although I understand it's probably an easier approach, although in the
>> end you probably have to filter it and end up just accepting what's
>> beneath the "test" from the grammar, no? (i.e.: that's what a lambda
>> body accepts).
>>
>
> Using PyParser_ASTFromString is the easiest possible way to do this. Given
> a string, it returns an AST node. What could be simpler?


​I think that for implementation purposes, given the python infrastructure,
it's fine, but for specification purposes, probably incorrect... As I don't
think f-strings should accept:

 f"start {import sys; sys.version_info[0];} end" (i.e.:
​
PyParser_ASTFromString doesn't just return an expression, it accepts any
valid Python code, even code which can't be used in an f-string).


> ​Well, I think all language implementations / IDEs (or at least those
>> which want to give syntax errors) will *have* to look inside f-strings.
>>
>
> While it's probably true that IDEs (and definitely language
> implementations) will want to parse f-strings, I think there are many more
> code scanners that are not language implementations or IDEs. And by being
> "just" regular strings with a new prefix, it's trivial to get any parser
> that doesn't care about the internal structure to at least see f-strings as
> normal strings.
>
> Also, you could still have a separate grammar saying how to look inside
>> f-strings (this would make the lives of other implementors easier) even
>> if it was a post-processing step as you're doing now.
>>
>
> Yes. I've contemplated exposing the f-string scanner. That's the part that
> returns expressions (as strings) and literal strings. I realize that won't
> help 3.6.


Nice...

As a note, just for the record, my own interest on f-strings is knowing how
exactly they are parsed for providing a preview of PyDev with syntax
highlighting and preliminary support for f-strings (which at the very
minimum besides syntax highlighting for the parts of f-strings should also
show syntax errors inside them).

​Cheers,​

​Fabio
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation

2016-11-04 Thread Eric V. Smith 

On 11/4/2016 2:03 PM, Fabio Zadrozny wrote:


Using PyParser_ASTFromString is the easiest possible way to do this.
Given a string, it returns an AST node. What could be simpler?


​I think that for implementation purposes, given the python
infrastructure, it's fine, but for specification purposes, probably
incorrect... As I don't think f-strings should accept:

 f"start {import sys; sys.version_info[0];} end" (i.e.:
​
PyParser_ASTFromString doesn't just return an expression, it accepts any
valid Python code, even code which can't be used in an f-string).


Not so. It should only accept expressions, not statements:

>>> f"start {import sys; sys.version_info[0];} end"
  File "", line 1
(import sys; sys.version_info[0];)
  ^
SyntaxError: invalid syntax


Also, you could still have a separate grammar saying how to look
inside
f-strings (this would make the lives of other implementors
easier) even
if it was a post-processing step as you're doing now.


Yes. I've contemplated exposing the f-string scanner. That's the
part that returns expressions (as strings) and literal strings. I
realize that won't help 3.6.


Nice...

As a note, just for the record, my own interest on f-strings is knowing
how exactly they are parsed for providing a preview of PyDev with syntax
highlighting and preliminary support for f-strings (which at the very
minimum besides syntax highlighting for the parts of f-strings should
also show syntax errors inside them).


I understand there's a need to make the specification more rigorous. 
Hopefully we'll get there.


Eric.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

2016-11-04 Thread Miquel Torres 
Nice! For the record, I'll be giving a talk in PyCon Ireland about
Codespeed. Would you mind me citing those tweets and screenshots, to
highlight usage on speed.python.org?

You mentioned new more reliable vs old results. How close are we to have an
stable setup that gives us benchmarks numbers regularly?

Cheers,
Miquel
El El vie, 4 nov 2016 a las 12:30, Victor Stinner 
escribió:

> Hi,
>
> You may know that I'm working on benchmarks. I regenerated all
> benchmark results of speed.python.org using performance 0.3.2
> (benchmark suite). I started to analyze results.
>
> All results are available online on the website:
>
>https://speed.python.org/
>
>
> To communicate on my work on benchmarks, I tweeted two pictures:
>
> "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python
> 2.7 #python #benchmark":
> https://twitter.com/VictorStinner/status/794289596683210760
>
> "Python 3.6 is between 25% and 54% slower than Python 2.7 in the
> following benchmarks":
> https://twitter.com/VictorStinner/status/794305065708376069
>
>
> Many people were disappointed that Python 3.6 can be up to 54% slower
> than Python 2.7. In fact, I know many reasons which explain that, but
> it's hard to summarize them in 140 characters ;-)
>
> For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark
> pycrypto_aes. This benchmark tests a pure Python implementation of the
> crypto cipher AES. You may know that CPython is slow for CPU intensive
> functions, especially on integer and floatting point numbers.
>
> "int" in Python 3 is now "long integers" by default, which is known to
> be a little bit slower than "short int" of Python 2. On a more
> realistic benchmark (see other benchmarks), the overhead of Python 3
> "long int" is negligible.
>
> AES is a typical example stressing integers. For me, it's a dummy
> benchmark: it doesn't make sense to use Python for AES: modern CPUs
> have an *hardware* implemention which is super fast.
>
>
> Well, I didn't have time to analyze in depth individual benchmarks. If
> you want to help me, here is the source code of benchmarks:
> https://github.com/python/performance/blob/master/performance/benchmarks/
>
>
> Raw results of Python 3.6 compared to Python 2.7:
> ---
> $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz
> 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5
> Slower (40):
> - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower
> - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x
> slower
> - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower
> - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower
> - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower
> - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower
> - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower
> - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower
> - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower
> - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower
> - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower
> - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower
> - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower
> - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower
> - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower
> - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower
> - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower
> - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower
> - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower
> - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower
> - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower
> - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower
> - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower
> - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower
> - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower
> - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower
> - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower
> - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower
> - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower
> - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower
> - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower
> - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower
> - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower
> - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower
> - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower
> - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower
> - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower
> - float: 290 ms +- 5 ms -> 310 m

Re: [Python-Dev] itertools predicates

2016-11-04 Thread Francisco Couzo 
Well sorry about it, I didn't know about that PEP 8 line, disregard this
thread then :)

On Fri, Nov 4, 2016 at 2:29 PM, Raymond Hettinger <
[email protected]> wrote:

>
> > On Nov 3, 2016, at 3:51 PM, Steven D'Aprano  wrote:
> >
> > Before there can be a solution, there first has to be a problem that
> > needs solving. "Lack of consistency" is not necessarily a problem. The
> > intertools functions are quite different, they do different things with
> > different APIs. The question here should not be "why don't these
> > functions take None as an argument?", rather it should be "why should
> > these functions take None as an argument?".
>
> I concur with Steven who articulated the issue perfectly.  There isn't an
> actual problem here that needs to be solved (i.e. not a single user report
> in 15 years indicating that the API wasn't meeting the needs for real
> use-cases).  I'm disinclined to churn the API unless there is a real need.
>
> FWIW, groupby() has the predicate as an optional argument so that you can
> write groupby('aaabbc') and have it group by value (much like the
> key-function on sorted() is optional).   The two filter variants allow None
> as the first argument only for historical reasons -- once "bool" came
> along, it would have been better to write filter(bool, someiterable) in
> preference to using None which is less clear about its intention.  The
> takewhile/dropwhile tools didn't have the same constraint to match a
> historical API, so there was an opportunity to have a clearer API with a
> simpler signature.
>
> As Terry suggested, if you have other itertools feature requests, please
> put them on the tracker an assign them to me.
>
> Thank you,
>
>
> Raymond
>
>
>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> franciscouzo%40gmail.com
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance

2016-11-04 Thread Victor Stinner 
2016-11-04 20:18 GMT+01:00 Miquel Torres :
> Nice! For the record, I'll be giving a talk in PyCon Ireland about
> Codespeed. Would you mind me citing those tweets and screenshots, to
> highlight usage on speed.python.org?

Sure. Keep me in touch in you publish your slides later.


> You mentioned new more reliable vs old results. How close are we to have an
> stable setup that gives us benchmarks numbers regularly?

My plan for the short term is to analyze last (latest?) benchmarks
hiccups and try to fix them.

The fully automated script to run benchmarks is already written:
https://github.com/python/performance/tree/master/scripts

Then, the plan we decided with Zachary Ware is to run a script in a
loop which compiles the default branch of CPython. Later, we may also
do the same for 2.7 and 3.6 branches. And then add PyPy (and PyPy 3).

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com