[Python-Dev] Subprocess destructor now logs a ResourceWarning

2016-08-17 Thread Victor Stinner 
Hi,

FYI I made a tiny enhancement in the subprocess module in Python 3.6:
Popen destructor now logs a ResourceWarning if the child process is
still running when the destructor is called. Keeping a child process
in background is bad because it is likely to create zombi process,
because Python will no more read its exit status.

Martin Panter asked to add a new "detach() method" to subprocess.Popen
if you would like to "transfer the ownership" of the child process
(pid) to a different object. I'm not sure that it is needed in
practice (Popen is responsible to handle different resources, not only
the pid, maybe I didn't look closely to the issue), but it may be
required for specific code to avoid the new ResourceWarning. Please
take a look at :

   https://bugs.python.org/issue27068

Another enhancement:  ResourceWarning now also logs where "leaked"
resource was created if you enable the tracemalloc module. Example
with tracemalloc logging 10 frames (truncated output for readability):
-
$ ./python -X tracemalloc=10 -m test test_sys test_os
Run tests sequentially
0:00:00 [1/2] test_sys
^C

.../Lib/subprocess.py:1023: ResourceWarning: subprocess 28856 is still running
  ResourceWarning, source=self)
Object allocated at (most recent call first):
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 702
universal_newlines=True)
  (...)

.../Lib/test/libregrtest/main.py:343: ResourceWarning: unclosed file
<_io.TextIOWrapper name=3 encoding='UTF-8'>
  break
Object allocated at (most recent call first):
  File "/home/haypo/prog/python/default/Lib/subprocess.py", lineno 955
self.stdout = io.TextIOWrapper(self.stdout)
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 702
universal_newlines=True)
  File "/home/haypo/prog/python/default/Lib/test/test_sys.py", lineno 714
out = self.c_locale_get_error_handler(encoding=':ignore')
  (...)

Test suite interrupted by signal SIGINT.
2 tests omitted:
test_os test_sys

Total duration: 551 ms
-

Please run tests using -Wd (enabled by default in debug builds) and
watch for ResourceWarning, and then report these warnings on the bug
tracker. Each ResourceWarning is likely to highlight a bug.

I just fixed the subprocess ResourceWarning in the
test.support.script_helper module (the child process is now killed on
error, and I added a context manager to cleanup all Popen resources).

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues in Python TLS

2016-08-17 Thread Nick Coghlan 
On 14 August 2016 at 09:14, Benjamin Peterson  wrote:
> Correctness of TLS certificate verification is known to depend deeply on
> distribution. Python began to verify certificates by default only in in
> version 2.7.9. Many OS distributions (in particular, Ubuntu) did not
> enable verification for their stable distributions for backwards
> compatibility reasons. You might find looking at distro bugs for
> CVE-2014-9365 edifying.

For RHEL (and hence CentOS),
https://access.redhat.com/articles/2039753 summarises the current
status.

Since the system Python in RHEL 7.x is nominally Python 2.7.5, the
default behaviour reflects that. However, both PEP 466 and PEP 476
have been backported (as of RHEL 7.2 and the corresponding CentOS
release), so requests picks up the new capabilities implicitly, while
the standard library can be configured for default verification by
writing:

   [https]
   verify=enable

to /etc/python/cert-verification.cfg

Since folks are now able to force verification *off* via that same
configuration file mechanism, we've also reserved the right to
eventually change the default Python standard library behaviour on a
fresh RHEL 7 installation to be to verify certificates.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available

2016-08-17 Thread Philipp A. 
hi, i already posted in python-ideas, but i got no replies, so i’ll post
here:

in essence i think interpreting escape sequences in f-literals is a *very*
bad idea, mainly because i think it’s fundamental that syntax highlighters
can highlight code as code.

so they should highlight the code parts of f-literals as code to avoid bugs
and misunderstandings (parts highlighted as string aren’t expected to
execute). for many syntax highlighters, it’s *impossible* to properly
highlight python code after this change makes python’s grammar recursive
(escaped code in f-literals, doubly-escaped code in escaped f-literals in
f-literals, …).

i want this changed because i think it’s only done this way to reuse the
string tokenization code (i.e. convenience), not for some deeper reason.
the RFC even says so. however, f-literals aren’t strings, but expressions,
so string tokenization rules shouldn’t apply to the non-string parts.

how am i going about changing f-literal grammar before the beta hits?

i consider this important enough to defer f-literals to 3.7 if it can’t get
in in time.

best, philipp

Ned Deily  schrieb am Di., 16. Aug. 2016 um 05:02 Uhr:

> On behalf of the Python development community and the Python 3.6 release
> team, I'm happy to announce the availability of Python 3.6.0a4. 3.6.0a4
> is the last of four planned alpha releases of Python 3.6, the next major
> release of Python. During the alpha phase, Python 3.6 remains under
> heavy development: additional features will be added and existing
> features may be modified or deleted. Please keep in mind that this is a
> preview release and its use is not recommended for production environments.
>
> You can find Python 3.6.0a4 here:
>
> https://www.python.org/downloads/release/python-360a4/
>
> The next planned release of Python 3.6 will be 3.6.0b1, currently
> scheduled for 2016-09-12. 3.6.0b1 will mark the beginning of the beta
> phase of Python 3.6; at that time, feature development for 3.6 will be
> complete and the emphasis will change to fixing bugs and regressions.
> More information about the release schedule can be found here:
>
> https://www.python.org/dev/peps/pep-0494/
>
> --Ned
>
> --
>   Ned Deily
>   [email protected] -- []
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available

2016-08-17 Thread Brett Cannon 
Please don't cross-post as it means anyone replying to your email will now
split the conversation as not everyone will be subscribed to all of the
mailing lists you sent this to. I have stripped out all but python-dev for
my reply.

On Wed, 17 Aug 2016 at 09:47 Philipp A.  wrote:

> hi, i already posted in python-ideas, but i got no replies, so i’ll post
> here:
>

I don't remember specifically seeing any email on this. Do you have a link
to your post from the python-ideas archive showing your email actually made
it to the list?


>
> in essence i think interpreting escape sequences in f-literals is a *very*
> bad idea, mainly because i think it’s fundamental that syntax highlighters
> can highlight code as code.
>

I believe you're referring to
https://www.python.org/dev/peps/pep-0498/#escape-sequences ?


>
> so they should highlight the code parts of f-literals as code to avoid
> bugs and misunderstandings (parts highlighted as string aren’t expected to
> execute). for many syntax highlighters, it’s *impossible* to properly
> highlight python code after this change makes python’s grammar recursive
> (escaped code in f-literals, doubly-escaped code in escaped f-literals in
> f-literals, …).
>
> i want this changed because i think it’s only done this way to reuse the
> string tokenization code (i.e. convenience), not for some deeper reason.
> the RFC even says so. however, f-literals aren’t strings, but expressions,
> so string tokenization rules shouldn’t apply to the non-string parts.
>
> how am i going about changing f-literal grammar before the beta hits?
>

You can post to python-ideas and start a discussion there as the PEP has
already been accepted and implemented with the current semantics or ask for
clarification for the reasoning behind the decision here on python-dev.


>
> i consider this important enough to defer f-literals to 3.7 if it can’t
> get in in time.
>

I just wanted to let you know, Philipp, that your email comes off as
somewhat demanding, e.g. "I want this changed". Had you asked why the
decision was made then your email would not come off as "I'm right and
you're wrong" and more about you asking for clarification to understand
why, and then if you still disagreed with the thought process then bring up
that you think it may have been a mistake.

-Brett


>
> best, philipp
>
> Ned Deily  schrieb am Di., 16. Aug. 2016 um 05:02 Uhr:
>
>> On behalf of the Python development community and the Python 3.6 release
>> team, I'm happy to announce the availability of Python 3.6.0a4. 3.6.0a4
>> is the last of four planned alpha releases of Python 3.6, the next major
>> release of Python. During the alpha phase, Python 3.6 remains under
>> heavy development: additional features will be added and existing
>> features may be modified or deleted. Please keep in mind that this is a
>> preview release and its use is not recommended for production
>> environments.
>>
>> You can find Python 3.6.0a4 here:
>>
>> https://www.python.org/downloads/release/python-360a4/
>>
>> The next planned release of Python 3.6 will be 3.6.0b1, currently
>> scheduled for 2016-09-12. 3.6.0b1 will mark the beginning of the beta
>> phase of Python 3.6; at that time, feature development for 3.6 will be
>> complete and the emphasis will change to fixing bugs and regressions.
>> More information about the release schedule can be found here:
>>
>> https://www.python.org/dev/peps/pep-0494/
>>
>> --Ned
>>
>> --
>>   Ned Deily
>>   [email protected] -- []
>>
>> ___
>>
> Python-Dev mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/flying-sheep%40web.de
>>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available

2016-08-17 Thread Philipp A. 
Brett Cannon  schrieb am Mi., 17. Aug. 2016 um 19:15 Uhr:

> Please don't cross-post as it means anyone replying to your email will now
> split the conversation as not everyone will be subscribed to all of the
> mailing lists you sent this to. I have stripped out all but python-dev for
> my reply.
>

sorry, i’ll remember that! i just hit reply on the post and didn’t realize
it was posted to more than python-dev.

I don't remember specifically seeing any email on this. Do you have a link
> to your post from the python-ideas archive showing your email actually made
> it to the list?
>

i posted here
…

I believe you're referring to
> https://www.python.org/dev/peps/pep-0498/#escape-sequences ?
>

yes!

how am i going about changing f-literal grammar before the beta hits?
>>
>
> You can post to python-ideas and start a discussion there as the PEP has
> already been accepted and implemented with the current semantics or ask for
> clarification for the reasoning behind the decision here on python-dev.
>

thanks. i’d like to hereby do the latter. i think the PEP’s wording is
pretty clear:

Due to Python's string tokenizing rules, the f-string f'abc {a['x']} def' is
> invalid. The tokenizer parses this as 3 tokens: f'abc {a[' , x , and ']}
> def' . Just like regular strings, this cannot be fixed by using raw
> strings. There are a number of correct ways to write this f-string
>

i guess that means that python’s string tokenization rules are reused for
f-literals, even though they aren’t actually strings. could someone please
explain if this is right and if so, why it was chosen to do this instead of
writing more fitting tokenization code?

naively i’d assume f'abc {a['x']} def' to tokenize as something like:

F_BEGIN
  F_STRING_BEGIN "a" "b" "c" " " F_STRING_END
  F_EXPR_START
NAME_START "a" NAME_END
GETITEM_BEGIN STRING_BEGIN "x" STRING_END GETITEM_END
  F_EXPR_END
  F_STRING_BEGIN " " "d" "e" "f" F_STRING_END
F_END

where f-literals are defined as F_START + F_STRING + (F_EXPR + F_STRING)* +
F_END

all of this of course accounting for different delimiters and so on

i consider this important enough to defer f-literals to 3.7 if it can’t get
>> in in time.
>>
>
> I just wanted to let you know, Philipp, that your email comes off as
> somewhat demanding, e.g. "I want this changed". Had you asked why the
> decision was made then your email would not come off as "I'm right and
> you're wrong" and more about you asking for clarification to understand
> why, and then if you still disagreed with the thought process then bring up
> that you think it may have been a mistake.
>

sorry. i just wanted to make my feelings clear since i think this is an
overlooked issue and the time is tight, in hope that maybe someone is
inspired to listen. i thought the PEP’s wording was hint enough to explain
the rationale (convenient reuse of tokenization code)

i’ll patiently await clarification about this, and again: sorry for
sounding demanding :(

-Brett
>

Cheers, Philipp
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available

2016-08-17 Thread Guido van Rossum 
On Wed, Aug 17, 2016 at 10:58 AM, Philipp A.  wrote:

> i’ll patiently await clarification about this, and again: sorry for
> sounding demanding :(
>

The explanation is honestly that the current approach is the most
straightforward for the implementation (it's pretty hard to intercept the
string literal before escapes have been processed) and nobody cares enough
about the edge cases to force the implementation to jump through more hoops.

I really don't think this discussion should be reopened. If you disagree,
please start a new thread on python-ideas. Everybody: please don't follow
up to python-dev.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] {RELEASE] Python 3.6.0a4 is now available

2016-08-17 Thread Brett Cannon 
On Wed, 17 Aug 2016 at 10:58 Philipp A.  wrote:

> Brett Cannon  schrieb am Mi., 17. Aug. 2016 um
> 19:15 Uhr:
>
>> Please don't cross-post as it means anyone replying to your email will
>> now split the conversation as not everyone will be subscribed to all of the
>> mailing lists you sent this to. I have stripped out all but python-dev for
>> my reply.
>>
>
> sorry, i’ll remember that! i just hit reply on the post and didn’t realize
> it was posted to more than python-dev.
>
> I don't remember specifically seeing any email on this. Do you have a link
>> to your post from the python-ideas archive showing your email actually made
>> it to the list?
>>
>
> i posted here
> …
>

That never made it to the mailing list because you posted it to Google
Groups (notice how it isn't in the April archives for the mailing list:
https://mail.python.org/pipermail/python-ideas/2016-April/thread.html).


>
> I believe you're referring to
>> https://www.python.org/dev/peps/pep-0498/#escape-sequences ?
>>
>
> yes!
>
> how am i going about changing f-literal grammar before the beta hits?
>>>
>>
>> You can post to python-ideas and start a discussion there as the PEP has
>> already been accepted and implemented with the current semantics or ask for
>> clarification for the reasoning behind the decision here on python-dev.
>>
>
> thanks. i’d like to hereby do the latter. i think the PEP’s wording is
> pretty clear:
>
> Due to Python's string tokenizing rules, the f-string f'abc {a['x']} def' is
>> invalid. The tokenizer parses this as 3 tokens: f'abc {a[' , x , and ']}
>> def' . Just like regular strings, this cannot be fixed by using raw
>> strings. There are a number of correct ways to write this f-string
>>
>
> i guess that means that python’s string tokenization rules are reused for
> f-literals, even though they aren’t actually strings.
>

They are still strings, there is just post-processing on the string itself
to do the interpolation.


> could someone please explain if this is right and if so, why it was chosen
> to do this instead of writing more fitting tokenization code?
>

My suspicion is simplification of the implementation, but Eric Smith can
tell me if I'm wrong. By doing it this way the implementation can use
Python itself to do the tokenizing of the string, while if you do the
string interpolation beforehand you would then need to do it entirely at
the C level which is very messy and painful since you're explicitly
avoiding Python's automatic handling of Unicode, etc. As I think you
pointed out, doing it the way it is currently implemented allows for
re-using the str.format() code which is way easier. When it comes to an
open source project where no one is paid to work on it, easy and pragmatic
beats out purer and harder (it's in the Zen of Python after all :) .

You also make it harder to work with Unicode-based variable names (or at
least explain it). If you have Unicode in a variable name but you can't use
\N{} in the string to help express it you then have to say "normal Unicode
support in the string applies everywhere *but* in the string interpolation
part".

Or another reason is you can explain f-strings as "basically
str.format_map(**locals(), **globals()), but without having to make the
actual method call" (and worrying about clashing keys but I couldn't think
of a way of using dict.update() in a single line). But with your desired
change it kills this explanation by saying f-strings aren't like this but
some magical string that does all of this stuff before normal string
normalization occurs.


>
> naively i’d assume f'abc {a['x']} def' to tokenize as something like:
>
> F_BEGIN
>   F_STRING_BEGIN "a" "b" "c" " " F_STRING_END
>   F_EXPR_START
> NAME_START "a" NAME_END
> GETITEM_BEGIN STRING_BEGIN "x" STRING_END GETITEM_END
>   F_EXPR_END
>   F_STRING_BEGIN " " "d" "e" "f" F_STRING_END
> F_END
>
> where f-literals are defined as F_START + F_STRING + (F_EXPR + F_STRING)*
> + F_END
>
> all of this of course accounting for different delimiters and so on
>
> i consider this important enough to defer f-literals to 3.7 if it can’t
>>> get in in time.
>>>
>>
>> I just wanted to let you know, Philipp, that your email comes off as
>> somewhat demanding, e.g. "I want this changed". Had you asked why the
>> decision was made then your email would not come off as "I'm right and
>> you're wrong" and more about you asking for clarification to understand
>> why, and then if you still disagreed with the thought process then bring up
>> that you think it may have been a mistake.
>>
>
> sorry. i just wanted to make my feelings clear since i think this is an
> overlooked issue and the time is tight, in hope that maybe someone is
> inspired to listen. i thought the PEP’s wording was hint enough to explain
> the rationale (convenient reuse of tokenization code)
>
> i’ll patiently await clarification about this, and again: sorry for
> sounding demandi