Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Cory Benfield 

> On 7 Jun 2017, at 20:06, Jim Baker  wrote:
> 
> Sorry, I wish this were true, but PyOpenSSL is not available on Jython, 
> because we do not yet support CFFI for Jython. CFFI support is something we 
> have looked at, but we have not implemented. (There is a related and far more 
> ambitious project to implement support for the C Extension API, 
> http://jyni.org/ , which Stefan Richthofer is working on 
> with me under the Google Summer of Code.)

This is what I was worried about. Moving to require PyOpenSSL *also* locks us 
out of Jython support, at least for the time being. That’s another point in the 
“con” column for making PyOpenSSL a mandatory dependency.

> I should mention that we sometimes see undocumented functionality leak out. 
> For example, 
> https://docs.python.org/3/library/socket.html#socket.socket.listen 
>  doesn't 
> say anything about backlog=0, but the requests test suite (last time I looked 
> on Jython) now depends on it. We assumed it was something like 
> http://pubs.opengroup.org/onlinepubs/009695399/functions/listen.html 
> , but 
> as described in http://bugs.python.org/issue8498 
> , it now means that "a server application 
> in python that accepts exactly 1 connection", by passing to the underlying C. 
> More tests, more docs, please, even though of course that's a lot of dev 
> effort.

Ah, yes, we do. In our defense, this is the semantic of the listen syscall, and 
the socket module is generally a pretty thin wrapper around most of the socket 
syscalls. But in hindsight this is definitely the kind of thing that gets 
tricky for someone trying to reimplement the socket module in terms of a 
different abstraction. I don’t want to dive down this rabbit hole because if we 
do I’ll have to start talking about how the complexity of the socket API is one 
of the other implementation hurdles for PEP 543, but I think that’s a 
conversation for another time.

Cory___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Victor Stinner 
2017-06-08 10:30 GMT+02:00 Cory Benfield :
> This is what I was worried about. Moving to require PyOpenSSL *also* locks
> us out of Jython support, at least for the time being. That’s another point
> in the “con” column for making PyOpenSSL a mandatory dependency.

Even if we do backport MemoryBIO to the next Python 2.7.14, I don't
think that you can require MemoryBIO. What about all existing
operating systems which provide a Python 2.7 without MemoryBIO? You
need to have a workaround anyway. For example, make the new
asynchronous API optional and use the old blocking mode in the
meanwhile.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Support for RFC 6920: Naming Things with Hashes?

2017-06-08 Thread Stephen J. Turnbull 
Sylvain Bellemare writes:
 > Hi,
 > 
 >  I hope this is the right place to post this kind of question. If not I
 > apologize.
 > 
 > I was simply wondering if anyone had been looking into supporting RFC 6920 (
 > https://tools.ietf.org/html/rfc6920).
 > 
 > For a simple example of what this is about see
 > https://tools.ietf.org/html/rfc6920#section-8

This is not the best place.  My first take would be "This is a great
idea!  But it doesn't need to be in core.  Write it up and post it to
PyPI."

My second would be "there's probably a special interest group (SIG)
mailing list" that's more appropriate.  See https://mail.python.org/;
WebSIG list looks like a good bet.  Third take, try
[email protected] and see what the general Python users say.
Many are web developers.

Finally, I am not a web expert, just a long time follower of
discussions on these lists.  If you are really sure this belongs in
the stdlib right now, you could try [email protected].  That
list is for proposing and refining new ideas, this one is more for
discussion of stuff that has an implementation in progress and needs
fine-tuning.

Happy hacking!

Steve

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Donald Stufft 

> On Jun 8, 2017, at 6:36 AM, Victor Stinner  wrote:
> 
> 2017-06-08 10:30 GMT+02:00 Cory Benfield :
>> This is what I was worried about. Moving to require PyOpenSSL *also* locks
>> us out of Jython support, at least for the time being. That’s another point
>> in the “con” column for making PyOpenSSL a mandatory dependency.
> 
> Even if we do backport MemoryBIO to the next Python 2.7.14, I don't
> think that you can require MemoryBIO. What about all existing
> operating systems which provide a Python 2.7 without MemoryBIO? You
> need to have a workaround anyway. For example, make the new
> asynchronous API optional and use the old blocking mode in the
> meanwhile.
> 

I mentioned it earlier, but using the current download numbers from PyPI, 
<2.7.9 is rapidly dropping and is likely going to be single digit %age within 
the next 6-8 months IIRC. If 2.7.14 follows a similar trajectory, requests can 
depend on it within like… 2 years? Maybe 3? Likely it will depend on whether 
2.7.14 gets into the next Ubuntu LTS or not.

—
Donald Stufft



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Victor Stinner 
Maybe the intent of my PEP is unclear: the goal is not to allow
Requests to require MemoryBIO, but to get a wide adoption of a future
implementation of the new TLS API (PEP). IMHO having an implementation
working on the latest Python 2.7 version should make it possible to
use it on some kinds of applications.

I should take time to read the last messages in this thread and try to
summarize them in the PEP ;-)

Victor

2017-06-08 13:06 GMT+02:00 Donald Stufft :
>
> On Jun 8, 2017, at 6:36 AM, Victor Stinner  wrote:
>
> 2017-06-08 10:30 GMT+02:00 Cory Benfield :
>
> This is what I was worried about. Moving to require PyOpenSSL *also* locks
> us out of Jython support, at least for the time being. That’s another point
> in the “con” column for making PyOpenSSL a mandatory dependency.
>
>
> Even if we do backport MemoryBIO to the next Python 2.7.14, I don't
> think that you can require MemoryBIO. What about all existing
> operating systems which provide a Python 2.7 without MemoryBIO? You
> need to have a workaround anyway. For example, make the new
> asynchronous API optional and use the old blocking mode in the
> meanwhile.
>
>
> I mentioned it earlier, but using the current download numbers from PyPI,
> <2.7.9 is rapidly dropping and is likely going to be single digit %age
> within the next 6-8 months IIRC. If 2.7.14 follows a similar trajectory,
> requests can depend on it within like… 2 years? Maybe 3? Likely it will
> depend on whether 2.7.14 gets into the next Ubuntu LTS or not.
>
> —
> Donald Stufft
>
>
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Antoine Pitrou 
On Thu, 8 Jun 2017 09:30:37 +0100
Cory Benfield  wrote:
> 
> Ah, yes, we do. In our defense, this is the semantic of the listen 
> syscall,[...]

According to POSIX, the backlog is only a hint, i.e. Jython is probably
ok in not observing its value:

"""The backlog argument provides a hint to the implementation which the
implementation shall use to limit the number of outstanding connections
in the socket's listen queue."""

http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Buildbot report, june 2017

2017-06-08 Thread Victor Stinner 
Hi,

With the help of Zachary Ware, we succeed to set up a mailing list
getting email notifications when a buildbot fails whereas it
previously succeeded (green/success or orange/warnings):
https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/

Please subscribe if you want to stay tuned.

I started to create bug reports for each buildbot failures on 2.7,
3.5, 3.6 and master (future 3.7) branches. I try to reply to buildbot
emails with a reference to the issue (I create a new issue if needed).
The good news is that these issues are being slowly fixed one by one!

In general, buildbots are now stable and at least able to catch the
obvious regressions.

Zachary Ware added two new "Refleaks" buildbot slaves, on Windows 8
and Gentoo, to catch reference leaks. We are fixing leaks. It seems
like some of them are recent. Previously, we add a server ran by
Antoine Pitrou, but Antoine didn't update its configuration for Git
(when CPython migrated from Mercurial). I like the idea of an
integrated "Refleaks" server in our buildbot farm.

Information on CPython buildbots:
https://www.python.org/dev/buildbot/

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Nick Coghlan 
On 8 June 2017 at 21:30, Victor Stinner  wrote:
> Maybe the intent of my PEP is unclear: the goal is not to allow
> Requests to require MemoryBIO, but to get a wide adoption of a future
> implementation of the new TLS API (PEP). IMHO having an implementation
> working on the latest Python 2.7 version should make it possible to
> use it on some kinds of applications.
>
> I should take time to read the last messages in this thread and try to
> summarize them in the PEP ;-)

I think the short version of the main unanswered questions would be:

1. How hard would it be to define a full wheel-installable
"backport.ssl" module that made the latest 3.x ssl module API
available to Python 2 applications?
2. Given such a module, how hard would it be to add an "install()"
function that added it to sys.modules as the "ssl" module?

While that wouldn't allow pip or requests to depend on it without some
solution to the bootstrapping problem, it would clearly separate the
two questions, and also give us an architecture closer to the way
importlib2 works (i.e. if you want to make that the implementation of
import in a Python 2 application, you have to explicitly enable it at
runtime, but you can also install it whenever you want, rather than
having to rely on your Python implementation to provide it).

We'd also still be left with the probably of figuring out how to
handle Jython, but I figure we can consider that part of the
bootstrapping problem: how to let applications reliably get hold of
"backport.ssl" under a name provided by the host Python 2.7
implementation, rather than the normal pip-installable name.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2017 Python Language Summit coverage finale

2017-06-08 Thread Jake Edge 

Hola python-dev,

I (finally!) finished up the coverage from the language summit ...

The starting point is the overview article, here:
https://lwn.net/Articles/723251/ which should now be free for anyone to
see (and the first seven articles too).  LWN subscribers can see the
articles right away, but one week after they are published in the weekly
edition, they become freely available for everyone.  Until then,
though, feel free to share the SubscriberLinks I am posting here.  I
have been asked about our policy on appropriate places to share
SubscriberLinks; blogs, tweets, social media, mailing lists, etc. are
all perfectly fine with us.

The new articles are:

Classes and types in the Python typing module
https://lwn.net/Articles/724639/ or
https://lwn.net/SubscriberLink/724639/b055ef642e22abfc/

Status of mypy and type checking https://lwn.net/Articles/724740/ or
https://lwn.net/SubscriberLink/724740/5d0b0158880097cf/

Language summit lightning talks https://lwn.net/Articles/723823/ or
https://lwn.net/SubscriberLink/723823/695e5554c4d68f1b/

Thanks again to Larry and Barry for inviting me!

jake

-- 
Jake Edge - LWN - [email protected] - http://lwn.net
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Steve Dower 
I'm just going to straight up admit that I've lost track of the point of 
this thread.


It sounds like we don't *need* to backport any of ssl into the Python 
2.7 standard library, as long as we can bundle a 3rd-party backport for pip?


I assume that, at a high level, the operation needed is to download 
content over https using the system trust stores. Is that what we're 
trying to achieve here?


If it is, can we just provide an enhanced urlretrieve()? Certainly on 
Windows, and presumably on macOS, it's much easier to do the high-level 
GET operation than to reimplement it using primitives.


As far as I can tell (bearing in mind that my brain can't keep track of 
this thread anymore), the only argument against this is if someone wants 
sensible default behaviour with local configuration tweaks: treat one 
specific certificate as valid while also treating the system (and user) 
certificate stores as valid without actually putting your certificate 
into the system (or user) store.


My gut reaction to that is to say "too bad - choices are 100% system 
configuration or 100% custom configuration". I suspect that's not a 
suitable reaction, but I can't explain why. (And bear in mind that the 
current state of the ssl module on Windows will never get 100% system 
configuration.)


Can someone explain to me why pip can't use a simple "system store 
urlretrieve()" function without configuration and "OpenSSL 
urlretrieve()" function with fully custom configuration? Even if only to 
bootstrap something that *can* merge two entirely different 
configuration systems?


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for RFC 6920: Naming Things with Hashes?

2017-06-08 Thread Sylvain Bellemare 
On 8 June 2017 at 12:37, Stephen J. Turnbull <
[email protected]> wrote:

> Sylvain Bellemare writes:
>  > Hi,
>  >
>  >  I hope this is the right place to post this kind of question. If not I
>  > apologize.
>  >
>  > I was simply wondering if anyone had been looking into supporting RFC
> 6920 (
>  > https://tools.ietf.org/html/rfc6920).
>  >
>  > For a simple example of what this is about see
>  > https://tools.ietf.org/html/rfc6920#section-8
>
> This is not the best place.  My first take would be "This is a great
> idea!  But it doesn't need to be in core.  Write it up and post it to
> PyPI."
>
> My second would be "there's probably a special interest group (SIG)
> mailing list" that's more appropriate.  See https://mail.python.org/;
> WebSIG list looks like a good bet.  Third take, try
> [email protected] and see what the general Python users say.
> Many are web developers.
>
> Finally, I am not a web expert, just a long time follower of
> discussions on these lists.  If you are really sure this belongs in
> the stdlib right now, you could try [email protected].  That
> list is for proposing and refining new ideas, this one is more for
> discussion of stuff that has an implementation in progress and needs
> fine-tuning.
>
> Happy hacking!
>
> Steve
>
>
Thanks for the detailed answer! I will follow your tips!
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Paul Moore 
On 8 June 2017 at 17:40, Steve Dower  wrote:
> I'm just going to straight up admit that I've lost track of the point of
> this thread.

You have my sympathies - I'm not really following it either :-(

> It sounds like we don't *need* to backport any of ssl into the Python 2.7
> standard library, as long as we can bundle a 3rd-party backport for pip?

My understanding is that the PEP is asking to backport a new feature.
The problem is that as a new feature, this received some (justifiable)
pushback.

The arguments for why this new feature is needed then got messy - as I
understand it, it's something to do with how the requests library
moves forward - they are blocked from supporting newer async features
from 3.x because they need to support 2.7. This feature would relieve
that logjam for them. Obviously, as a 3rd party library, their issues
aren't particularly compelling for the stdlib - but because pip uses
requests, and pip is shipped with Python, things get complicated.

> I assume that, at a high level, the operation needed is to download content
> over https using the system trust stores. Is that what we're trying to
> achieve here?
>
> If it is, can we just provide an enhanced urlretrieve()? Certainly on
> Windows, and presumably on macOS, it's much easier to do the high-level GET
> operation than to reimplement it using primitives.

The problem is that pip uses more features of requests than just
issuing GET requests. We aren't going to be in a position to switch to
a simple urlretrieve operation, even as some sort of fallback. What
I'm personally not at all clear on is why we can't just ship a version
of pip that supports 2.7 with 2.7, and a later version with 3.x. That
doesn't make the problem for pip and requests any easier, but it does
make it not python-dev's problem.

The issue is that the gulf between 2.7 and 3.x is getting wider, and
it's starting to cause real pain to 3rd party projects like requests.
Does that justify backporting this specific feature to 2.7? I don't
know.

Note that I haven't actually read the original PEP. I don't have a
view on networking issues, security, or Python 2.7 support. So I
didn't really feel the need to more than skim this thread. My only
interest really is where pip gets involved - and that's where I get
confused as I don't really see why (ensure)pip makes the problem so
much more complicated.

Paul

PS I'd be amazed if my summary above isn't wrong in at least some key
points. Corrections welcome!
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Donald Stufft 

> On Jun 8, 2017, at 12:40 PM, Steve Dower  wrote:
> 
> I'm just going to straight up admit that I've lost track of the point of this 
> thread.
> 
> It sounds like we don't *need* to backport any of ssl into the Python 2.7 
> standard library, as long as we can bundle a 3rd-party backport for pip?
> 
> I assume that, at a high level, the operation needed is to download content 
> over https using the system trust stores. Is that what we're trying to 
> achieve here?


The basic yak stak here is:

* PEP 543 should be the future, it is a much much better way of handling TLS 
than our current ssl module is.
* Cory can’t spend his work time on PEP 543 unless he can say it is useful for 
requests.
* In order for PEP 543 to be useful for requests, he needs a way to provide a 
backport for it for Python 2.7.
   * This backport *CAN* be OpenSSL only, but needs to be able to provide the 
same API.
* PEP 543 wants to work with MemoryBIOs instead of sockets, because a MemoryBio 
is a much much better way of implementing this problem for a variety of 
reasons, and it would be a mistake to use a socket primitive again.
* Indepently, requests also wants to be able to provide the ability for people 
to use it with asyncio, however it can’t drop support for Python 2.7 in the 
quest for doing that. Twisted provides a way forward that lets requests work on 
both 2.x and 3.x and integrate with asyncio, but Twisted requires MemoryBio to 
do so.
* pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for both 
cases from up above, however, pip cannot depend on a C library that isn’t part 
of the standard library - in addition this would break alternative runtimes 
like Jython where pyOpenSSL doesn’t work.

Thus, adding MemoryBio to 2.7’s ssl means that requests can use it instead of 
depending on a C package (which it can’t because of pip), which means that Cory 
can then justify working on PEP 543 as part of his requests work, because he 
can say on Python 3.7, We use PEP 543 natively, and on Python < 3.7 we either 
no longer support that Python or we can wrap ssl.MemoryBio using a pure Python 
backport shim that provides the same API as PEP 543.

Indendently of that, adding MemoryBio to 2.7’s ssl means that Twisted can use 
it instead of depending on a C package, which means requests can depend on 
Twisted (which it otherwise can’t, again because of pip). This then means that 
requests can refactor itself internally to use Twisted to write asyncio 
compatible code that provides a synchronous API and a asynchronous API that 
works on 2.7 and 3.x with asyncio. All of the other options require effectively 
forking the code or the ecosystem by either having “this library you use for 
sync” and “the library you use for async” largely duplicating code OR requires 
all of the network libraries to drop support for 2.7 (can’t, most of their 
users are on 2.7 still) or requires forking the library to have a 2.x and a 3.x 
version (we tried this before early on in the 3.x split and we settled on the 
fact that a single code base is a much better way to handle straddling the 
2.x/3.x line).

So basically back porting MemoryBio unlocks two important things for the health 
of the Python ecosystem:

* Allows forward progress on PEP 543, which provides a wealth of great benefits 
like using the platform trust model and removing the need for pip, requests, 
etc to bundle a CA bundle internally and removing the need (long term anyways) 
for Python to ship a copy of OpenSSL on platforms that don’t provide it.
* Allows requests and other libraries to continue to straddle the 2.x/3.x line 
where they need to, while still providing people who are using Python 3.x a way 
to use asyncio without having to fork the entire ecosystem into having an aio* 
copy of every single network library that exists.


> 
> Can someone explain to me why pip can't use a simple "system store 
> urlretrieve()" function without configuration and "OpenSSL urlretrieve()" 
> function with fully custom configuration? Even if only to bootstrap something 
> that *can* merge two entirely different configuration systems?

It would require rewriting significant parts of our code that interfaces with 
HTTP, including the fact that we rely on some additional requests libraries 
(like cache control) to implement HTTP caching in pip, so unless urlretireve 
supported that as well we’d have to completely rewrite that from scratch.


—
Donald Stufft



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Donald Stufft 

> On Jun 8, 2017, at 3:37 PM, Donald Stufft  wrote:
> 
> So basically back porting MemoryBio unlocks two important things for the 
> health of the Python ecosystem:
> 
> * Allows forward progress on PEP 543, which provides a wealth of great 
> benefits like using the platform trust model and removing the need for pip, 
> requests, etc to bundle a CA bundle internally and removing the need (long 
> term anyways) for Python to ship a copy of OpenSSL on platforms that don’t 
> provide it.
> * Allows requests and other libraries to continue to straddle the 2.x/3.x 
> line where they need to, while still providing people who are using Python 
> 3.x a way to use asyncio without having to fork the entire ecosystem into 
> having an aio* copy of every single network library that exists.


Sorry I forgot one other important benefit:

* It reduces the delta between the 3.x and the 2.x ssl and _ssl modules, which 
makes actually maintaining those modules easier because this code is fiddly and 
hard to get right, so the more we can just directly backport security fixes 
from one to the other rather than having to rewrite the patch, the better off 
we are.

And the downside here is pretty small honestly?

It’s not changing behavior of anything that currently exists since it’s adding 
a new thing inside the ssl module and Alex has already written the patch so 
there’s little extra work to do and it actually makes maintenance easier since 
patches can more readily be applied straight from `master`.

The primary argument I can see against it, is one of purity, that 2.7 shouldn’t 
get new features but as we know, practicality beats purity ;) (and we’ve 
already accepted that TLS is a special case, special enough to break the rules, 
so the main question is whether this specific thing is worthwhile— which given 
it’s benefits to the Python ecosystem and to maintaining Python, I think it is).

—
Donald Stufft



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Steve Dower 

On 08Jun2017 1237, Donald Stufft wrote:


The basic yak stak here is:

* PEP 543 should be the future, it is a much much better way of handling 
TLS than our current ssl module is.
* Cory can’t spend his work time on PEP 543 unless he can say it is 
useful for requests.
* In order for PEP 543 to be useful for requests, he needs a way to 
provide a backport for it for Python 2.7.
* This backport *CAN* be OpenSSL only, but needs to be able to 
provide the same API.
* PEP 543 wants to work with MemoryBIOs instead of sockets, because a 
MemoryBio is a much much better way of implementing this problem for a 
variety of reasons, and it would be a mistake to use a socket primitive 
again.
* Indepently, requests also wants to be able to provide the ability for 
people to use it with asyncio, however it can’t drop support for Python 
2.7 in the quest for doing that. Twisted provides a way forward that 
lets requests work on both 2.x and 3.x and integrate with asyncio, but 
Twisted requires MemoryBio to do so.
* pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for 
both cases from up above, however, pip cannot depend on a C library that 
isn’t part of the standard library - in addition this would break 
alternative runtimes like Jython where pyOpenSSL doesn’t work.


Awesome, this is exactly what I needed to see.

What if Python 2.7 just exposed the OpenSSL primitives necessary so that 
ctypes could use them? Is that a viable approach here? Presumably then a 
MemoryBIO backport could be pure-Python.


It doesn't help the other *ythons, but unless they have MemoryBIO 
implementations ready to backport then I can't think of anything that 
will help them short of a completely new API.


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Donald Stufft 

> On Jun 8, 2017, at 3:57 PM, Steve Dower  wrote:
> 
> On 08Jun2017 1237, Donald Stufft wrote:
>> The basic yak stak here is:
>> * PEP 543 should be the future, it is a much much better way of handling TLS 
>> than our current ssl module is.
>> * Cory can’t spend his work time on PEP 543 unless he can say it is useful 
>> for requests.
>> * In order for PEP 543 to be useful for requests, he needs a way to provide 
>> a backport for it for Python 2.7.
>>* This backport *CAN* be OpenSSL only, but needs to be able to provide 
>> the same API.
>> * PEP 543 wants to work with MemoryBIOs instead of sockets, because a 
>> MemoryBio is a much much better way of implementing this problem for a 
>> variety of reasons, and it would be a mistake to use a socket primitive 
>> again.
>> * Indepently, requests also wants to be able to provide the ability for 
>> people to use it with asyncio, however it can’t drop support for Python 2.7 
>> in the quest for doing that. Twisted provides a way forward that lets 
>> requests work on both 2.x and 3.x and integrate with asyncio, but Twisted 
>> requires MemoryBio to do so.
>> * pyOpenSSL *could* be used to provide the MemoryBio needed on 2.7 for both 
>> cases from up above, however, pip cannot depend on a C library that isn’t 
>> part of the standard library - in addition this would break alternative 
>> runtimes like Jython where pyOpenSSL doesn’t work.
> 
> Awesome, this is exactly what I needed to see.
> 
> What if Python 2.7 just exposed the OpenSSL primitives necessary so that 
> ctypes could use them? Is that a viable approach here? Presumably then a 
> MemoryBIO backport could be pure-Python.
> 
> It doesn't help the other *ythons, but unless they have MemoryBIO 
> implementations ready to backport then I can't think of anything that will 
> help them short of a completely new API.
> 


I would have to let Cory answer the question about feasibility here since he’s 
much more familiar with OpenSSL’s API (and even binding something like this 
with ctypes) than I am. The first thing that really stands out to me though is 
it just feels a bit like shuffling deckchairs?

At least I don’t see what adding the new feature of “thing you can use ctypes 
with to get a MemoryBio” does that adding the ssl.MemoryBio feature doesn’t 
other than downsides:

* It creates yet another divergence between 3.x and 2.x that makes it harder to 
maintain ssl and _ssl.
* It’s harder to use for requests/Twisted/etc.
* It’s not already written (Ok this is minor, but still!).

What do you see the benefits of that approach being over just backporting 
ssl.MemoryBio?

—
Donald Stufft



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

2017-06-08 Thread Cory Benfield 

> On 8 Jun 2017, at 21:17, Donald Stufft  wrote:
> 
>> 
>> On Jun 8, 2017, at 3:57 PM, Steve Dower > > wrote:
>> 
>> Awesome, this is exactly what I needed to see.
>> 
>> What if Python 2.7 just exposed the OpenSSL primitives necessary so that 
>> ctypes could use them? Is that a viable approach here? Presumably then a 
>> MemoryBIO backport could be pure-Python.
>> 
>> It doesn't help the other *ythons, but unless they have MemoryBIO 
>> implementations ready to backport then I can't think of anything that will 
>> help them short of a completely new API.
>> 
> 
> 
> I would have to let Cory answer the question about feasibility here since 
> he’s much more familiar with OpenSSL’s API (and even binding something like 
> this with ctypes) than I am. The first thing that really stands out to me 
> though is it just feels a bit like shuffling deckchairs?

The short answer is that, while it’s do-able, we have some problems with ABI 
compatibility. OpenSSL 1.1 and 1.0 are ABI incompatible, so I have to write 
divergent ctypes code to handle each case. It may also be relevant to support 
OpenSSL 0.9.x so we roll into the same ABI compatibility concern all over 
again. Doubly annoyingly a bunch of OpenSSL code in 1.0 is actually macros 
which don’t work in ctypes so there’ll be a lot of futzing about in structures 
to get what I need to do done.

This also doesn’t get into the difficulty of some operating systems shipping a 
LibreSSL masquerading as an OpenSSL, which is subtly incompatible in ways I 
don’t fully understand at this time.

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Extracting python bytecode from a linux core dump?

2017-06-08 Thread Steve Dower 
I can’t help with the gdb commands, but I’d suggest you want to start from one 
of the global variables for the sys module, probably the modules dict. You’ll 
have to reverse engineer the memory structures to find its values and each of 
their dicts, eventually finding function objects pointing to code objects which 
will point to bytecode blobs. All of these structures are in the sources, so it 
shouldn’t be that hard, just time consuming (I’ve done it on Windows before 
with different tools).

If you know the code was being executed when the dump was made you could look 
at the stack to find calls in the EvalFrame function. Those should have a local 
or argument to the code object or bytecode (my memory on names and structures 
in 2.6 isn’t that good).

A final alternative would be to find the function type object address and 
search memory for that to locate function objects and then code objects. That 
might be the best one, if you can locate the type object in the dump.

Hope that helps,
Steve

Top-posted from my Windows phone

From: CompuTinker
Sent: Wednesday, June 7, 2017 13:26
To: [email protected]
Subject: [Python-Dev] Extracting python bytecode from a linux core dump?

I hope this is the proper place for internals questions...

I have a core file (produced via the gcore command) of a linux python2.6 
process.  I need to extract the byte code and de-compile it.

I looked at https://wiki.python.org/moin/DebuggingWithGdb and related pages.  
However, these all seem to require either a running process and/or a binary 
with debugging symbols.

I'm thinking that the compiled bytecode is likely in an array or contiguous set 
of memory within the python executable's image and that there's probably a way 
to pull it out with gdb.  Unsurprisingly, the pyc 0xd1f20d0a magic number isn't 
kept in memory. So, how do I find the memory holding the compiled byte-code ?

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] IMPORTANT: Python 3.6.2 Maintenance Release Release Candidate in 3+ days (Monday 2017-06-12 12:00 UTC)

2017-06-08 Thread Ned Deily 
We are approaching the end of the second calendar quarter of 2017 and, 
according to PEP 494, it's time to start producing the second maintenance 
release for the 3.6 series.  The schedule calls for the release candidate to be 
produced on Monday 2017-06-12 UTC.  As was the case with previous 3.6.x 
releases, the plan is for the release candidate to be the same as the final 
release, that is, no additional changes go in after the release candidate 
except for any showstopper critical problems that might be discovered with rc1. 
 So please plan to get any security fixes, bug fixes, and documentation changes 
you think should be in 3.6.2 merged in ASAP.  The 3.6.2 final is planned for 
two weeks following rc1, that is, on 2017-06-26.  The next 3.6 maintenance 
release (3.6.3) is planned to follow about 3 months later, so most likely in 
2017-09.

A reminder that the 3.6 branch will remain open for checkins throughout the 
3.6.2rc and final cycle.  Once 3.6.2rc1 is tagged, new commits to the 3.6 
branch will release with 3.6.3.  As always, if you find any problem in 3.6.2rc1 
that you may believe should be corrected in 3.6.2, please ensure the problem is 
documented in a new or existing open issue on bugs.python.org, ensure that the 
Priority field of the issue is set to "release blocker", and that "Python 3.6" 
is included in the selected Versions.  Comments or tags on github Pull Requests 
are NOT sufficient!

Thanks again for all of your efforts in bringing 3.6 into the world and for 
helping now to make it even better!

https://www.python.org/dev/peps/pep-0494/
http://cpython-devguide.readthedocs.io

--
  Ned Deily
  [email protected] -- []

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com