Re: [Python-Dev] PEP 7 and braces { .... } on if

2017-06-05 Thread Stefan Behnel 
Serhiy Storchaka schrieb am 03.06.2017 um 18:25:
> Yet about braces. PEP 7 implicitly forbids breaking the line before an
> opening brace. An opening brace should stay at the end the line of the
> outer compound statement.
> 
> if (mro != NULL) {
> ...
> }
> else {
> ...
> }
> 
> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
> type->tp_dictoffset == b_size &&
> (size_t)t_size == b_size + sizeof(PyObject *)) {
> return 0; /* "Forgive" adding a __dict__ only */
> }
> 
> But the latter example continuation lines are intended at the same level as
> the following block of code. I propose to make exception for that case and
> allow moving an open brace to the start of the next line.
> 
> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
> type->tp_dictoffset == b_size &&
> (size_t)t_size == b_size + sizeof(PyObject *))
> {
> return 0; /* "Forgive" adding a __dict__ only */
> }
> 
> This adds a visual separation of a multiline condition from the following
> code.

Python itself has a similar problem and solves it differently. Why not take
a look at PEP-8 here?

"""
Yes:

# Aligned with opening delimiter.
foo = long_function_name(var_one, var_two,
 var_three, var_four)

# More indentation included to distinguish this from the rest.
def long_function_name(
var_one, var_two, var_three,
var_four):
print(var_one)

# Hanging indents should add a level.
foo = long_function_name(
var_one, var_two,
var_three, var_four)

No:

# Arguments on first line forbidden when not using vertical alignment.
foo = long_function_name(var_one, var_two,
var_three, var_four)

# Further indentation required as indentation is not distinguishable.
def long_function_name(
var_one, var_two, var_three,
var_four):
print(var_one)
"""

ISTM that overindenting the conditions (i.e. following the "more
indentation" example) solves this problem and makes it very readable. The
same can be done in C code and avoids having to remember two different
special ways to do it for core devs in C and Python.

Stefan

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Cory Benfield 

> On 2 Jun 2017, at 17:39, Nick Coghlan  wrote:
> 
> On 3 June 2017 at 02:22, Donald Stufft  wrote:
>> It’s not just bootstrapping that pip has a problem with for C extensions, it
>> also prevents upgrading PyOpenSSL on Windows because having pip import
>> PyOpenSSL locks the .dll, and we can’t delete it or overwrite it until the
>> pip process exits and no longer imports PyOpenSSL. This isn’t a problem on
>> Linux or macOS or the other *nix clients though. We patch requests as it is
>> today to prevent it from importing simplejson and cryptography for this
>> reason.
> 
> Would requests be loading PyOpenSSL on Windows, though? If the aim is
> to facilitate PEP 543, then I'd expect it to be using the SChannel
> backend in that case.

Only assuming the SChannel backend is available. This would also likely be a 
third-party Python library so you can just search and replace “PyOpenSSL” above 
with “SChannel” and hit the exact same problem.

Cory
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Cory Benfield 

> On 2 Jun 2017, at 17:59, Donald Stufft  wrote:
> 
> I suspect (though I’d let him speak for himself) that Cory would rather 
> continue to be sync only than require pip to go back to not using requests.

We are not wedded to supporting pip, but I think the interaction between the 
two tools is positive. I think pip gains a lot from depending on us, and we get 
a lot of value from having pip as a dependency. So the cost of supporting pip 
would have to be pretty darn high for us to want to stop doing that, and so on 
this issue I think Donald is right.

Cory___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Cory Benfield 

> On 3 Jun 2017, at 07:25, Nick Coghlan  wrote:
> 
> As a result, as awkward and annoying as it would be, I'm wondering if
> the structure of the requests migration may need to be something like:
> 
> - switch unconditionally to an async backend on Py3
> - switch conditionally to an async backend on Py2 when PyOpenSSL is available
> - retain sufficient sync-only support on Py2 to allow pip to disable
> the PyOpenSSL dependency

Ultimately we don’t have the resources to do this.

The requests core development team is 4 people, 3 of whom are part time 
best-effort maintainers and one of whom is me, who has foolishly decided to 
also work on PEP 543 as well as take over the lead maintenance role on urllib3. 
urllib3 has another two additional best-effort maintainers.

We simply do not have the development resources to support two parallel 
branches of code. Even if we did, doing so is a recipe for adding bugs and 
inconsistencies between the two, leading to increased workload as we break our 
users and fail to keep the two branches in sync. It also makes it much harder 
for users to migrate from our synchronous backend to our async one, as those 
two no longer use identical underlying implementations and so will have subtle 
inconsistencies.

The TL;DR there is: no, we’d rather stay sync only than do that.

Cory

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Nick Coghlan 
On 5 June 2017 at 18:42, Cory Benfield  wrote:
> On 3 Jun 2017, at 07:25, Nick Coghlan  wrote:
> As a result, as awkward and annoying as it would be, I'm wondering if
> the structure of the requests migration may need to be something like:
>
> - switch unconditionally to an async backend on Py3
> - switch conditionally to an async backend on Py2 when PyOpenSSL is
> available
> - retain sufficient sync-only support on Py2 to allow pip to disable
> the PyOpenSSL dependency
>
> Ultimately we don’t have the resources to do this.
>
> The requests core development team is 4 people, 3 of whom are part time
> best-effort maintainers and one of whom is me, who has foolishly decided to
> also work on PEP 543 as well as take over the lead maintenance role on
> urllib3. urllib3 has another two additional best-effort maintainers.
>
> We simply do not have the development resources to support two parallel
> branches of code. Even if we did, doing so is a recipe for adding bugs and
> inconsistencies between the two, leading to increased workload as we break
> our users and fail to keep the two branches in sync. It also makes it much
> harder for users to migrate from our synchronous backend to our async one,
> as those two no longer use identical underlying implementations and so will
> have subtle inconsistencies.
>
> The TL;DR there is: no, we’d rather stay sync only than do that.

Would you be OK with the notion of a "just for pip bootstrapping"
private backend in _ensurepip_ssl?

That is, the only officially supported async-backed requests
configuration on Py2 would be with the PyOpenSSL dependency installed,
but in collaboration with the pip devs we'd also plumb in the pieces
to let a new async-backed requests work without any extension modules
other than those in the standard library.

That way, the only thing truly gated on the backport would be *pip*
updating its bundled version of requests to the async-backed version -
for normal third party use, the change would be "you need PyOpenSSL",
rather than "you need a newer version of Python".

We'd still effectively end up with two different code execution paths
(one backed by PyOpenSSL, one backed by the new private _ensurepip_ssl
extension module), but the level of divergence would be much lower
(it's just a question of where MemoryBIO and SSLObject are coming
from) and the support scope for the less frequently used path would be
much narrower (i.e. if a problem report isn't related to pip
bootstrapping, it can be closed as "won't fix")

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Cory Benfield 

> On 5 Jun 2017, at 12:00, Nick Coghlan  wrote:
> 
> Would you be OK with the notion of a "just for pip bootstrapping"
> private backend in _ensurepip_ssl?
> 
> That is, the only officially supported async-backed requests
> configuration on Py2 would be with the PyOpenSSL dependency installed,
> but in collaboration with the pip devs we'd also plumb in the pieces
> to let a new async-backed requests work without any extension modules
> other than those in the standard library.
> 
> That way, the only thing truly gated on the backport would be *pip*
> updating its bundled version of requests to the async-backed version -
> for normal third party use, the change would be "you need PyOpenSSL",
> rather than "you need a newer version of Python".
> 
> We'd still effectively end up with two different code execution paths
> (one backed by PyOpenSSL, one backed by the new private _ensurepip_ssl
> extension module), but the level of divergence would be much lower
> (it's just a question of where MemoryBIO and SSLObject are coming
> from) and the support scope for the less frequently used path would be
> much narrower (i.e. if a problem report isn't related to pip
> bootstrapping, it can be closed as "won't fix”

It’s not clear to me what the structure of that looks like, or what work is 
required to achieve it.

Right now Twisted never uses MemoryBIO and SSLObject: it always uses PyOpenSSL. 
That seems like we’d need to add MemoryBIO and SSLObject support to Twisted 
which can be enabled in some way other than feature detection (that is, so it 
can be installed). Without PEP 543 this is pretty gross. With PEP 543 it sucks 
less, but it also gates this work behind PEP 543 being successfully implemented 
and landed.

I guess we are *open* to that approach? It’s not clear to me how beneficial 
that is, and it doesn’t gain any of the ecosystem benefits (no-one else on Py 2 
can ever use this chunk of tested, understood code), but it’s certainly an 
option. The indirection gives me pause though.

Cory___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Donald Stufft 

> On Jun 5, 2017, at 7:00 AM, Nick Coghlan  wrote:
> 
> On 5 June 2017 at 18:42, Cory Benfield  > wrote:
>> On 3 Jun 2017, at 07:25, Nick Coghlan  wrote:
>> As a result, as awkward and annoying as it would be, I'm wondering if
>> the structure of the requests migration may need to be something like:
>> 
>> - switch unconditionally to an async backend on Py3
>> - switch conditionally to an async backend on Py2 when PyOpenSSL is
>> available
>> - retain sufficient sync-only support on Py2 to allow pip to disable
>> the PyOpenSSL dependency
>> 
>> Ultimately we don’t have the resources to do this.
>> 
>> The requests core development team is 4 people, 3 of whom are part time
>> best-effort maintainers and one of whom is me, who has foolishly decided to
>> also work on PEP 543 as well as take over the lead maintenance role on
>> urllib3. urllib3 has another two additional best-effort maintainers.
>> 
>> We simply do not have the development resources to support two parallel
>> branches of code. Even if we did, doing so is a recipe for adding bugs and
>> inconsistencies between the two, leading to increased workload as we break
>> our users and fail to keep the two branches in sync. It also makes it much
>> harder for users to migrate from our synchronous backend to our async one,
>> as those two no longer use identical underlying implementations and so will
>> have subtle inconsistencies.
>> 
>> The TL;DR there is: no, we’d rather stay sync only than do that.
> 
> Would you be OK with the notion of a "just for pip bootstrapping"
> private backend in _ensurepip_ssl?
> 


Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip? 
Thinking about this reminded me about the *other* reason pip avoids 
dependencies— avoiding making assertions about what versions of software can 
actually be installed. IOW if pip depends on pyOpenSSL >= X.Y, then you 
essentially can’t install any other version of pyOpenSSL and you’d be forced to 
upgrade.

This isn’t end of the world and pyOpenSSL is generally stable enough we could 
*probably* get away with a unversioned dependency on a non Windows platform. On 
Windows we’d have to uh, I guess use a SChannel c-types backend? Not having our 
TLS library in the stdlib makes it a bit more difficult, but from pip’s own POV 
if there’s a MemoryBio that we’re allowed to use and then requests uses 
pyOpenSSL normally (we’d apply a small patch to pip’s copy of requests to make 
it use it, but that’s easy to do with our setup) I think that would be workable.

I think that’s a less optimal solution than just accepting PEP 546 which I 
think is beneficial to the Python ecosystem as a whole, making it easier to 
port more code onto 3.x (and allowing unlocking 3.x’s capabilities) and will 
make it easier to maintain the ssl library on 2.7, given it will have less of a 
divergence from 3.x again.


—
Donald Stufft



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 7 and braces { .... } on if

2017-06-05 Thread Skip Montanaro 
On Mon, Jun 5, 2017 at 12:41 AM, Serhiy Storchaka  wrote:
> Barry and Victor prefer moving a brace on a new line in all multiline
> conditional cases. I think that it should be done only when the condition
> continuation lines and the following block of the code have the same
> indentation (as in the example above), and the following code is enough
> readable:
>
> if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
>  "invalid escape sequence '\\%c'",
>  *first_invalid_escape) < 0) {
> Py_DECREF(result);
> return NULL;
> }
>
> What other core developers think about this?

Wow, this discussion takes me back. Glad I don't have to check out
comp.lang.c to get my brace placement fix. 

Skip
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Nick Coghlan 
On 5 June 2017 at 21:44, Donald Stufft  wrote:
> Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip?

Yes, so something like _tls_bootstrap would probably be a better name
for the helper module (assuming we go down the "private API to
bootstrap 3rd party SSL modules" path).

The leading underscore would be akin to the one on _thread - it isn't
that the API would be undocumented or inaccessible, it's that it
wouldn't have any backwards compatibility guarantees for general use,
so you shouldn't use it without a really good reason.

> Thinking about this reminded me about the *other* reason pip avoids
> dependencies— avoiding making assertions about what versions of software can
> actually be installed. IOW if pip depends on pyOpenSSL >= X.Y, then you
> essentially can’t install any other version of pyOpenSSL and you’d be forced
> to upgrade.

Indeed, and I think that's sufficient justification for at least
adding the private TLS bootstrapping API.

> This isn’t end of the world and pyOpenSSL is generally stable enough we
> could *probably* get away with a unversioned dependency on a non Windows
> platform. On Windows we’d have to uh, I guess use a SChannel c-types
> backend? Not having our TLS library in the stdlib makes it a bit more
> difficult, but from pip’s own POV if there’s a MemoryBio that we’re allowed
> to use and then requests uses pyOpenSSL normally (we’d apply a small patch
> to pip’s copy of requests to make it use it, but that’s easy to do with our
> setup) I think that would be workable.
>
> I think that’s a less optimal solution than just accepting PEP 546 which I
> think is beneficial to the Python ecosystem as a whole, making it easier to
> port more code onto 3.x (and allowing unlocking 3.x’s capabilities) and will
> make it easier to maintain the ssl library on 2.7, given it will have less
> of a divergence from 3.x again.

Right, I'm just trying to make sure we clearly separate the two
arguments: pip bootstrapping (which can potentially be addressed
through a private alternate SSL/TLS API) and ecosystem advancement
(making it easier for SSL/TLS capable applications to retain Python
2.7.x compatibility at least until 2020, while still supporting the
improvements in asynchronous IO capabilities in the 3.x series).

Separating the two main options like that gives us two future scenarios:

1. Private TLS bootstrapping API

- requests & any other libraries that want this functionality would
need a dependency on PyOpenSSL for Python versions prior to 3.5
- the base SSL/TLS support in Python 2.7 remains focused on
synchronous IO, with only limited support for sans-io style protocol
library implementations
- some time in 2017 or 2018, pip starts requiring that environments
provide either an externally managed _tls_bootstrap module, *or* a
sufficiently recent PyOpenSSL
- the first CPython maintenance release to include that pip update
would also provide the required _tls_bootstrap module by default
- redistributors will be able to backport _tls_bootstrap as an
independent module (e.g. as part of their system pip packages) rather
than necessarily including it directly in their Python runtime
- this distribution model should be resilient over time, allowing
_tls_bootstrap to be rebased freely without risking breaking end user
applications that weren't expecting it (this approach would likely
even be useful in LTS 3.x environments as they age - e.g. Ubuntu 14.04
and 16.04)
- the private _tls_bootstrap API would either be a straight backport
of the 3.6 ssl module, or else a shim around the corresponding
standard library's ssl module to add the required features, whichever
was judged easiest to implement and maintain

2. Public standard library ssl module API update

- libraries needing the functionality *either* depend on PyOpenSSL
*or* set their minimum Python version to 2.7.14+
- if libraries add a Python version check to their installation
metadata, LTS distributions are out of luck, since we typically don't
rebase Python (e.g. the RHEL 7 system Python still advertises itself
as 2.7.5, even though assorted backports have been applied on top of
that)
- even if libraries rely on API feature detection rather than explicit
version checks, redistributors still need to actually patch the
standard library - we can't just add a guaranteed-side-effect-free
module as a separate post-installation step
- while some system administrators may be willing and able to deploy
their own variant a _tls_boostrap module in advance of their
redistributors providing one, relatively few are going to be willing
to apply custom patches to Python before deploying it

So honestly, I don't think the current "ecosystem advancement"
argument in PEP 546 actually works, as the enhancement is
*significantly* easier to roll out to older 2.7.x releases if the
adoption model is "Supporting SSL/TLS with sans-io style protocol
implementations on Python versions prior to Python 3.5 requires the
use of PyOpenSSL o

Re: [Python-Dev] PEP 7 and braces { .... } on if

2017-06-05 Thread Ethan Furman 

 if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
  "invalid escape sequence '\\%c'",
  *first_invalid_escape) < 0) {
 Py_DECREF(result);
 return NULL;
 }

What other core developers think about this?


I would format that as:

 if (PyErr_WarnFormat(
 PyExc_DeprecationWarning,
 1,
 "invalid escape sequence '\\%c'",
 *first_invalid_escape) < 0)
 {
 Py_DECREF(result);
 return NULL;
 }

Because:

- having all the arguments on separate lines means
  - the function and first argument don't get run together
  - it's easy to pick out the individual arguments
- having the opening brace on its own line means
  - a little extra white space to buffer the condition and the body
  - it's easier to read the function name and then drop down to the
body

All in all, it becomes easier for (at least my) suboptimal eyes to read the 
code.

--
~Ethan~

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 7 and braces { .... } on if

2017-06-05 Thread MRAB 

On 2017-06-05 13:00, Skip Montanaro wrote:

On Mon, Jun 5, 2017 at 12:41 AM, Serhiy Storchaka  wrote:

Barry and Victor prefer moving a brace on a new line in all multiline
conditional cases. I think that it should be done only when the condition
continuation lines and the following block of the code have the same
indentation (as in the example above), and the following code is enough
readable:

if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
 "invalid escape sequence '\\%c'",
 *first_invalid_escape) < 0) {
Py_DECREF(result);
return NULL;
}

What other core developers think about this?


Wow, this discussion takes me back. Glad I don't have to check out
comp.lang.c to get my brace placement fix. 

FWIW, I half-indent continuation lines (an advantage of using spaces 
instead of tabs).

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-05 Thread Gregory P. Smith 
On Fri, Jun 2, 2017 at 12:33 PM Larry Hastings  wrote:

>
> On 06/02/2017 02:46 AM, Victor Stinner wrote:
>
> I would be curious of another test: use pymalloc for objects larger
> than 512 bytes. For example, allocate up to 4 KB?
>
> In the past, we already changed the maximum size from 256 to 512 to
> support most common Python objects on 64-bit platforms. Since Python
> objects contain many pointers: switching from 32 bit to 64 bit can
> double the size of the object in the worst case.
>
>
> You've already seen Tim Peters' post about why we must leave pool size set
> to 4k.  Obviously This in turn means using obmalloc for larger objects will
> mean more and more wasted memory.
>
> For example, let's say we use obmalloc for allocations of 2048 bytes.
> Pool size is 4096 bytes, and there's a 48-byte "pool_header" structure on
> the front (on 64-bit platforms, if I counted right).  So there are only
> 4048 bytes usable per pool.  After the first 2048 allocation, we're left
> with 2000 bytes at the end.  You can't use that memory for another
> allocation class, that's impossible given obmalloc's design.  So that 2000
> bytes is just wasted.
>
> Currently obmalloc's maximum allocation size is 512 bytes; after 7
> allocations, this leaves 464 wasted bytes at the end.  Which isn't *great*
> exactly but it's only 11% of the overall allocated memory.
>
> Anyway, I'm not super excited by the prospect of using obmalloc for larger
> objects.  There's an inverse relation between the size of allocation and
> the frequency of allocation.  In Python there are lots of tiny allocations,
> but fewer and fewer as the size increases.  (A similarly-shaped graph to
> what retailers call the "long tail".)  By no small coincidence, obmalloc is
> great at small objects, which is where we needed the help most.  Let's
> leave it at that.
>
>
> A more fruitful endeavor might be to try one of these fancy new
> third-party allocators in CPython, e.g. tcmalloc, jemalloc.  Try each with
> both obmalloc turned on and turned off, and see what happens to performance
> and memory usage.  (I'd try it myself, but I'm already so far behind on
> watching funny cat videos.)
>

FYI - in CPython using a different malloc instead of CPython's own obmalloc
is effectively a simple addition of ~three lines of code to something
Py_InitializeEx()-ish

thanks to tracemalloc:

PyMemAllocator pma;
...
PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &pma);
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &pma);

That sets the object allocator (normally obmalloc) to be the "system"
malloc which I assume you are having your alternate-malloc (tcmalloc,
jemalloc?, etc..) override.

As to where exactly I'd have to walk through the code... I see that it was
just refactored beyond what I used to recognize as part of cleaning up
interpreter startup.  (yay!) :)

For CPython at large, I don't want us to be in the business of shipping a
malloc implementation (any more than we already do). But it does seem worth
making what we've got tune-able without having to recompile to do it.

I liked the environment variable arena size setting idea. But there are
likely more things that could be offered up for tuning by people deploying
applications who have tested things to determine what is best for their
specific needs.

A note about OS/HW page sizes: While hugepages and transparent hugepages
*can* be a large performance increase due to less TLB cache misses, they
don't fit every application. They can be painful for anyone who depends on
a fork()'ed workers application model as a copy on write for a 2mb or 1gb
page due to a single refcount update is... costly.

-gps
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 544: Protocols - second round

2017-06-05 Thread Guido van Rossum 
On Fri, Jun 2, 2017 at 3:10 PM, Ivan Levkivskyi 
wrote:

> On 1 June 2017 at 00:10, Guido van Rossum  wrote:
>
>>
>> On Wed, May 31, 2017 at 2:16 AM, Ivan Levkivskyi 
>> wrote:
>>
>>> On 31 May 2017 at 00:58, Guido van Rossum  wrote:
>>> [...]
>>>
>>> Thank you for very detailed answers! I have practically nothing to add.
>>> It seems to me that most of the Kevin's questions stem from unnecessary
>>> focus
>>> on runtime type checking. Here are two ideas about how to fix this:
>>>
>>> * Add the word "static" somewhere in the PEP title.
>>>
>>
>> So the title could become "Protocols: Static structural subtyping (duck
>> typing)" -- long, but not record-setting.
>>
>
> I am thinking about "Protocols: Structural subtyping (static duck
> typing)". The reason is that subtyping is already a mostly static concept
> (in contrast to subclassing),
> while duck typing is typically associated with the runtime behaviour.
>
> This might seem minor, but this version of the title sounds much more
> naturally to me.
>

+1

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-05 Thread Larry Hastings 


On 06/04/2017 01:18 PM, Tim Peters wrote:

[Larry Hastings ]

...
Yet CPython's memory consumption continues to grow.  By the time a current
"trunk" build of CPython reaches the REPL prompt it's already allocated 16
arenas.

I'd be surprised if that's true ;-)  The first time `new_arena()` is
called, it allocates space for a vector of 16 (INITIAL_ARENA_OBJECTS)
`arena_object` structs.  Those are tiny, and hold bookkeeping info for
the actual arenas, none of which are allocated at first.


Oh!  I thought it also allocated the arenas themselves, in a loop.  I 
thought I saw that somewhere.  Happy to be proved wrong...



So at most 9 arenas ("highwater mark") were ever simultaneously allocated..


... though not completely off-base.


On 06/04/2017 11:50 AM, Tim Peters wrote:

I was hoping to spur a discussion of much higher level issues.  I bet
Larry was too ;-)


Actually I was hoping everyone would just tell me how right I was and 
thank me for my profound insights.



//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Nathaniel Smith 
On Jun 5, 2017 7:01 AM, "Nick Coghlan"  wrote:

On 5 June 2017 at 21:44, Donald Stufft  wrote:
> Is pip allowed to use the hypothetical _ensurepip_ssl outside of
ensurepip?

Yes, so something like _tls_bootstrap would probably be a better name
for the helper module (assuming we go down the "private API to
bootstrap 3rd party SSL modules" path).


It seems like there's a risk here that we end up with two divergent copies
of ssl.py and _ssl.c inside the python 2 tree, and that this will make it
harder to do future bug fixing and backports. Is this going to cause
problems? Is there any way to mitigate them?

-n
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE

2017-06-05 Thread Tim Peters 
[Larry]
> ...
> Oh!  I thought it also allocated the arenas themselves, in a loop.  I
> thought I saw that somewhere.  Happy to be proved wrong...

There is a loop in `new_arena()`, but it doesn't do what a casual
glance may assume it's doing ;-)  It's actually looping over the
newly-allocated teensy arena descriptor structs, linking them in to a
freelist and recording that they're _not_ (yet) associated with any
address space.


[Tim]
>> So at most 9 arenas ("highwater mark") were ever simultaneously allocated 
>> [by the
>> time the REPL prompt appeared in a 64-bit 3.6.1]..

> ... though not completely off-base.

Yes, 9 is in the ballpark of 16.

>> I was hoping to spur a discussion of much higher level issues.  I bet
>> Larry was too ;-)

> Actually I was hoping everyone would just tell me how right I was and thank
> me for my profound insights.

Thank you!  It should be thought about again.

I think _some_ increase of arena size should be a no-brainer, but I
don't expect it to help a lot.  For reasons already partly explained,
I expect we'd get much better bang for the buck by increasing the pool
size:

- Roughly speaking, we bash into a slows-the-code pool boundary 64
times as frequently as an arena boundary. If the arena size increased,
that ratio only gets worse.

- On 64-bit boxes the bytes lost to pool headers increased, but the
pool size did not.  Thus we're guaranteed to "waste" a higher
_percentage_ of allocated bytes than we did on a 32-bit box.

- The small object threshold also doubled.  Generally (not always),
the bytes lost to quantization increase the larger the size class.
For example, for the largest size class now, on a 64-bit box we can
only fit 7 512-byte objects into the 4096 - 48 = 4048 bytes that
remain in a pool.  So, in addition to the 48 pool header bytes, we
_also_ lose the 464 leftover bytes.  So we've lost 512 of the 4096
bytes:  12.5%.  That's certainly not typical, but in any case
quantization losses as percentage of total bytes decrease the larger
the pool.

Alas, I haven't thought of a plausibly practical way to replace
Py_ADDRESS_IN_RANGE unless the pool size increases a whole frickin'
lot :-(
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-05 Thread Nick Coghlan 
On 6 June 2017 at 10:59, Nathaniel Smith  wrote:
> On Jun 5, 2017 7:01 AM, "Nick Coghlan"  wrote:
>
> On 5 June 2017 at 21:44, Donald Stufft  wrote:
>> Is pip allowed to use the hypothetical _ensurepip_ssl outside of
>> ensurepip?
>
> Yes, so something like _tls_bootstrap would probably be a better name
> for the helper module (assuming we go down the "private API to
> bootstrap 3rd party SSL modules" path).
>
>
> It seems like there's a risk here that we end up with two divergent copies
> of ssl.py and _ssl.c inside the python 2 tree, and that this will make it
> harder to do future bug fixing and backports. Is this going to cause
> problems? Is there any way to mitigate them?

Aye, I spent some time thinking about potentially viable
implementation architectures, and realised that any "private API"
style solution pretty much *has* to be structured as a monkeypatching
API to be effective. Otherwise there is too much risk of divergence in
things like exception definitions that end up causing cryptic "Why is
my SSL exception handler not catching my SSL connection error?" type
bugs.

The gist of the approach would be that for libraries and
non-bootstrapped applications, their SSL/TLS feature detection code
would ultimately look something like this:

try:
# Don't do anything special if we don't need to
   from ssl import MemoryBIO, SSLObject
expect ImportError:
# Otherwise fall back to using PyOpenSSL
try:
from OpenSSL.SSL import Connection
except ImportError:
raise ImportError("Failed to import asynchronous SSL/TLS
support: ")

It's currently more complex than that in practice (since PyOpenSSL
doesn't natively offer an emulation of the affected parts of the
standard library's ssl module API), but that's what it would
effectively boil down to at the lowest level of any compatibility
wrapper.

Conveniently, this is *already* what libraries and non-bootstrapped
modules have to do if they're looking to support older Python versions
without requiring PyOpenSSL on newer releases, so there wouldn't
actually be any changes on that front.

By contrast, for bootstrapped applications (including pip) the
oversimplified compatibility import summary would instead look more
like this:

try:
# Don't do anything special if we don't need to
from ssl import MemoryBIO, SSLObject
expect ImportError:
# See if we can do a runtime in-place upgrade of the standard library
try:
import _tls_bootstrap
except ImportError:
# Otherwise fall back to using PyOpenSSL
try:
from OpenSSL.SSL import Connection
except ImportError:
raise ImportError("Failed to bootstrap asynchronous
SSL/TLS support: ")
else:
_tls_bootstrap.monkeypatch_ssl()

The reason this kind of approach is really attractive to
redistributors from a customer risk management perspective is that
like gevent's monkeypatching of synchronous networking APIs, it's
*opt-in at runtime*, so the risk of our accidentally inflicting it on
a customer that doesn't want it and doesn't need it is almost exactly
zero - if none of their own code includes the "import _tls_bootstrap;
_tls_bootstrap.monkeypatch_ssl()" invocation and none of their
dependencies start enabling it as an implicit side effect of some
other operation, they'll never even know the enhancement is there.
Instead, the compatibility risks get concentrated in the applications
relying on the bootstrapping API, since the monkeypatching process is
a potentially new source of bugs that don't exist in the more
conventional execution models.

The reason that's still preferable though, is that it means the
monkeypatching demonstrably doesn't have to be 100% perfect: it just
has to be close enough to the way the Python 3 standard library API
works to meet the needs of the applications that actually use it.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com