Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote: > The reason this kind of approach is really attractive to > redistributors from a customer risk management perspective is that > like gevent's monkeypatching of synchronous networking APIs, it's > *opt-in at runtime*, so the risk of our accidentally inflicting it on > a customer that doesn't want it and doesn't need it is almost exactly > zero - if none of their own code includes the "import _tls_bootstrap; > _tls_bootstrap.monkeypatch_ssl()" invocation and none of their > dependencies start enabling it as an implicit side effect of some > other operation, they'll never even know the enhancement is there. > Instead, the compatibility risks get concentrated in the applications > relying on the bootstrapping API, since the monkeypatching process is > a potentially new source of bugs that don't exist in the more > conventional execution models. OK. To unpack that, I think it would mean: 2.7's ssl.py and _ssl.c remain exactly as they are. We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start out as ~copies of the current ssl.py and _ssl.c code in master. (Unfortunately the SSLObject code can't just be separated out from _ssl.c into a new file, because the implementations of SSLObject and SSLSocket are heavily intertwined.) Then the copied files are hacked up to work in this monkeypatching context (so that e.g. instead of defining the SSLError hierarchy in C code, it gets imported from the main ssl.py). So: we end up with two copies of the ssl code in the py27 tree, both diverged from the copy that in the py3 tree, in different ways. I know that the ssl maintainers won't be *happy* about this given that keeping the py2 and py3 code similar is an ongoing issue and was one of the stated advantages of backporting SSLObject, but I don't know whether it's a showstopper or not; it'd be good to hear their perspective. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
With Nick's suggestion of _tls_bootstrap, this has certainly become more complicated. I'm still thinking of the ramifications for future Jython 2.7 support, if Python dev goes down this path. It still seems easier to me to avoid exposing the SSLObject/MemoryBIO model to 2.7 and instead concentrate on just having native TLS support. Yes, this might require more work than a simple backport, but everyone is resource constrained, not just the Requests or Jython teams. My concrete suggestion is that any work on PEP 543 will benefit from improving the testing currently found in test_ssl as part of its implementation. For a variety of reasons, test functions like ssl_io_loop ( https://github.com/python/cpython/blob/master/Lib/test/test_ssl.py#L1691) avoid asserting more than it can properly manage the framing of wrapped/unwrapped data, so for example one doesn't want to state explicitly when SSL_ERROR_WANT_READ would be called (too much white box at that point). On the other hand, the lack of unit testing of, for example, SSLObject.read, but instead only doing at the functional test level, means that there's nothing explicit testing "will raise an SSLWantReadError if it needs more data than the incoming BIO has available" (per the docs), or similar use of signaling (SSLEOFError comes to mind). The problem we have seen with Jython supporting similar functionality in the past in socket/select/ssl is not the happy path aspects like what ssl_io_loop tests, but underdocumented/undertested error states. So what benefits SChannel support, for example, should benefit Jython 3's implementation that would use Java's counterpart to SSLObject, SSLEngine. That would be a very good outcome of these proposed PEPs. - Jim On Tue, Jun 6, 2017 at 4:08 AM, Nathaniel Smith wrote: > On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote: > > The reason this kind of approach is really attractive to > > redistributors from a customer risk management perspective is that > > like gevent's monkeypatching of synchronous networking APIs, it's > > *opt-in at runtime*, so the risk of our accidentally inflicting it on > > a customer that doesn't want it and doesn't need it is almost exactly > > zero - if none of their own code includes the "import _tls_bootstrap; > > _tls_bootstrap.monkeypatch_ssl()" invocation and none of their > > dependencies start enabling it as an implicit side effect of some > > other operation, they'll never even know the enhancement is there. > > Instead, the compatibility risks get concentrated in the applications > > relying on the bootstrapping API, since the monkeypatching process is > > a potentially new source of bugs that don't exist in the more > > conventional execution models. > > OK. To unpack that, I think it would mean: > > 2.7's ssl.py and _ssl.c remain exactly as they are. > > We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start > out as ~copies of the current ssl.py and _ssl.c code in master. > (Unfortunately the SSLObject code can't just be separated out from > _ssl.c into a new file, because the implementations of SSLObject and > SSLSocket are heavily intertwined.) Then the copied files are hacked > up to work in this monkeypatching context (so that e.g. instead of > defining the SSLError hierarchy in C code, it gets imported from the > main ssl.py). > > So: we end up with two copies of the ssl code in the py27 tree, both > diverged from the copy that in the py3 tree, in different ways. > > I know that the ssl maintainers won't be *happy* about this given that > keeping the py2 and py3 code similar is an ongoing issue and was one > of the stated advantages of backporting SSLObject, but I don't know > whether it's a showstopper or not; it'd be good to hear their > perspective. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jbaker%40zyasoft.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The untuned tunable parameter ARENA_SIZE
On Mon, Jun 5, 2017 at 8:10 PM, Tim Peters wrote:
> [Tim]
> >> So at most 9 arenas ("highwater mark") were ever simultaneously
> allocated [by the
> >> time the REPL prompt appeared in a 64-bit 3.6.1]..
>
> > ... though not completely off-base.
>
> Yes, 9 is in the ballpark of 16.
>
>
> I think _some_ increase of arena size should be a no-brainer,
Maybe big enough to hold a bare, just started up interpreter?
> but I
> don't expect it to help a lot.
I was wondering about that -- in my experience with making re-sizable numpy
arrays, (not the same use-case, I know!), I found that allocating memory
wasn't all that slow, really. IN that use case, if you re-allocated every
time you added a single element, it was dog-slow. But once you allocated
less than, say, about every 10 or so, it started to make little difference
how much you over-allocated.
In this case, I'm thinking that as long as there is a not-tiny arena, it
just isn't going to make that much difference.
But only profiling will tell us.
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
[email protected]
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 and braces { .... } on if
On Jun 05, 2017, at 08:41 AM, Serhiy Storchaka wrote:
>the example above), and the following code is enough readable:
>
> if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
> "invalid escape sequence '\\%c'",
> *first_invalid_escape) < 0) {
> Py_DECREF(result);
> return NULL;
> }
My rationale for placing the opening brace on a separate line, indented to
under the `if` instead of hanging is that it's easier to miss the opening
brace in the example you posted above. Visually I (we?) tend to have a harder
time recognizing characters sitting way out to the right.
On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote:
>I would format that as:
>
> if (PyErr_WarnFormat(
> PyExc_DeprecationWarning,
> 1,
> "invalid escape sequence '\\%c'",
> *first_invalid_escape) < 0)
> {
> Py_DECREF(result);
> return NULL;
> }
In this case I'd *still* indent the opening brace to under the `if`. The
mismatched indentation between the open and close braces is jarring to me.
>- having all the arguments on separate lines means
> - the function and first argument don't get run together
> - it's easy to pick out the individual arguments
That's fine with me, but so is hanging the arguments, so I'd tend to leave
this up to the individual devs.
>- having the opening brace on its own line means
> - a little extra white space to buffer the condition and the body
> - it's easier to read the function name and then drop down to the
> body
Agreed with the rationale for the open brace being on a separate line, but did
you mean to indent the opening and closing braces to different levels?
Cheers,
-Barry
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 and braces { .... } on if
On Jun 05, 2017, at 07:00 AM, Skip Montanaro wrote: >Wow, this discussion takes me back. Glad I don't have to check out >comp.lang.c to get my brace placement fix. Life is better without braces! -Barry ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 and braces { .... } on if
On 07/06/2017 01:30, Barry Warsaw wrote:
On Jun 05, 2017, at 08:41 AM, Serhiy Storchaka wrote:
the example above), and the following code is enough readable:
if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1,
"invalid escape sequence '\\%c'",
*first_invalid_escape) < 0) {
Py_DECREF(result);
return NULL;
}
My rationale for placing the opening brace on a separate line, indented to
under the `if` instead of hanging is that it's easier to miss the opening
brace in the example you posted above. Visually I (we?) tend to have a harder
time recognizing characters sitting way out to the right.
On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote:
I would format that as:
if (PyErr_WarnFormat(
PyExc_DeprecationWarning,
1,
"invalid escape sequence '\\%c'",
*first_invalid_escape) < 0)
{
Py_DECREF(result);
return NULL;
}
In this case I'd *still* indent the opening brace to under the `if`. The
mismatched indentation between the open and close braces is jarring to me.
FWIW I feel exactly the same.
Rob Cliffe
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
On 6 June 2017 at 20:08, Nathaniel Smith wrote:
> On Mon, Jun 5, 2017 at 8:49 PM, Nick Coghlan wrote:
>> The reason this kind of approach is really attractive to
>> redistributors from a customer risk management perspective is that
>> like gevent's monkeypatching of synchronous networking APIs, it's
>> *opt-in at runtime*, so the risk of our accidentally inflicting it on
>> a customer that doesn't want it and doesn't need it is almost exactly
>> zero - if none of their own code includes the "import _tls_bootstrap;
>> _tls_bootstrap.monkeypatch_ssl()" invocation and none of their
>> dependencies start enabling it as an implicit side effect of some
>> other operation, they'll never even know the enhancement is there.
>> Instead, the compatibility risks get concentrated in the applications
>> relying on the bootstrapping API, since the monkeypatching process is
>> a potentially new source of bugs that don't exist in the more
>> conventional execution models.
>
> OK. To unpack that, I think it would mean:
>
> 2.7's ssl.py and _ssl.c remain exactly as they are.
>
> We create a new _tls_bootstrap.py and __tls_bootstrap.c, which start
> out as ~copies of the current ssl.py and _ssl.c code in master.
> (Unfortunately the SSLObject code can't just be separated out from
> _ssl.c into a new file, because the implementations of SSLObject and
> SSLSocket are heavily intertwined.) Then the copied files are hacked
> up to work in this monkeypatching context (so that e.g. instead of
> defining the SSLError hierarchy in C code, it gets imported from the
> main ssl.py).
>
> So: we end up with two copies of the ssl code in the py27 tree, both
> diverged from the copy that in the py3 tree, in different ways.
If we went down this path, I think the only way to keep it
maintainable would be to do one of two things:
1. Do the refactoring in Py3 first, but with the namespace pairing
being "ssl/_enable_ssl_backport" rather than "_tls_bootstrap/ssl"
2. Don't try to share any implementation details with Py2, and instead
provide a full "_ssl_backport" module more along the lines of
importlib2
With the second option, we'd just do a wholesale backport of the 3.6
SSL module under a different name, and projects like pip would use it
as follows:
# See if there is an updated SSL module backport available
try:
import _ssl_backport
except ImportError:
pass
else:
_ssl_backport.install() # Fails if ssl or _ssl were already imported
# At this point, sys.modules["ssl"] and sys.modules["_ssl"]
would match the
# API of the Python 3.6 versions, *not* the Python 2.7 versions.
# At the first 2.7.x maintenance release after 3.7, they'd be updated
# to match 3.7, and then the same again for 3.8.
# (3.9 is likely to be after the end of 2.7 maintenance)
# After that, proceed as usual with feature-based checks
try:
from ssl import MemoryBIO, SSLObject
expect ImportError:
# Otherwise fall back to using PyOpenSSL
try:
from OpenSSL.SSL import Connection
except ImportError:
raise ImportError("Failed to bootstrap asynchronous
SSL/TLS support: ")
Since there's no chance of breaking applications that don't opt in,
while still making the newer network security features available to
customers that want them, I think I could make a reasonable case for
backporting such an "_ssl_backport.install()" implementation into the
RHEL 7 system Python, but even if I wasn't able to do that, this kind
of runtime injection into sys.modules could still be shipped as an
add-on library in a way that an updated standard library SSL module
can't.
Such an approach would make the 2.7 regression tests noticeably slower
(since we'd need to re-run a subset of them with the backport being
injected by regrtest before anything loads the ssl module), and means
backporting SSL fixes to 2.7 would become a two step process (once to
resync the _ssl_backport module, and the second to update the default
implementation), but I think there would be enough benefits to make
that worth the pain:
- we'd have a baseline 2.7 SSL/TLS implementation that we can readily
keep in line with its 3.x counterparts
- this approach has a reasonable chance of making it through the
review processes for long term support distributions
- system and application integrators can more readily do their own
backports independently of redistributors
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 and braces { .... } on if
On 06/06/2017 05:30 PM, Barry Warsaw wrote:
On Jun 05, 2017, at 08:19 AM, Ethan Furman wrote:
I would format that as:
if (PyErr_WarnFormat(
PyExc_DeprecationWarning,
1,
"invalid escape sequence '\\%c'",
*first_invalid_escape) < 0)
{
Py_DECREF(result);
return NULL;
}
In this case I'd *still* indent the opening brace to under the `if`. The
mismatched indentation between the open and close braces is jarring to me.
- having all the arguments on separate lines means
- the function and first argument don't get run together
- it's easy to pick out the individual arguments
That's fine with me, but so is hanging the arguments, so I'd tend to leave
this up to the individual devs.
- having the opening brace on its own line means
- a little extra white space to buffer the condition and the body
- it's easier to read the function name and then drop down to the
body
Agreed with the rationale for the open brace being on a separate line, but did
you mean to indent the opening and closing braces to different levels?
It's what I see. Left to my own devices I would leave the opening brace where it is and indent the closing brace to
match. That way when I see code at the same level as the opening `if` I know I'm out of that block.
--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
