[Python-Dev] Suggested changes to verify HTTPS by default (was Re: Proposed schedule for 3.4.2)

2014-09-09 Thread Nick Coghlan 
On 9 September 2014 03:44, Alex Gaynor  wrote:
> *Shifts uncomfortably* it looks like presently there's not a good way to
> change anything about the SSL configuration for urllib.request.urlopen. It
> does not take a `context` argument, as the http.client API does:
> https://docs.python.org/3/library/urllib.request.html#module-urllib.request
> and instead takes the cafile, capath, cadefault args.
>
> This would need to be updated first, once it *did* take such an argument,
> this would be accomplished by:
>
> context = ssl.create_default_context()
> context.verify_mode = CERT_OPTIONACERT_NONE
> context.verify_hostname = False
> urllib.request.urlopen("https://something-i-apparently-dont-care-much-about";,
> context=context)

I'd never needed to use the existing global configuration settings in
urllib.request before, but it actually *does* already support setting
the default opener for urllib.urlopen.

To explicitly set it to use verified HTTPS by default:

import ssl, urllib.request
https_handler = HTTPSHandler(context=ssl.create_default_context(),
check_hostname=True)
urllib.request.install_opener(urllib.request.build_opener(https_handler)

When the default changes, turning off verification by default for
urllib.request.urlopen would look like:

import ssl, urllib.request
unverified_context = ssl.create_default_context()
unverified_context.verify_mode = CERT_OPTIONACERT_NONE
unverified_context.verify_hostname = False
unverified_handler = HTTPSHandler(context=unverified_context,
check_hostname=False)

urllib.request.install_opener(urllib.request.build_opener(unverified_handler)

However, even setting the opener like that still leaves
http.client.HTTPSConnection, urllib.request.URLOpener and
urllib.request.FancyURLOpener using unverified HTTPS with no easy way
to change their default behaviour.

That means some other approach to global configuration is going to be
needed to cover the "coping with legacy corporate infrastructure"
case, and I still think a monkeypatching based hack is likely to be
our best bet.

So, focusing on 3.4, but in a way that should be feasible to backport,
the changes that I now believe would be needed are:

1. Add "context" arguments to urlopen, URLOpener and FancyURLOpener
(the latter two have been deprecated since 3.3, but this would make
things easier for a subsequent 2.7 backport)
2. Add a ssl._create_https_context() alias for ssl.create_default_context()
3. Change urllib.request.urlopen() and http.client.HTTPSConnection to
call ssl_create_https_context() rather than
ssl._create_stdlib_context()
4. Rename ssl._create_stdlib_context() to
ssl._create_unverified_context() (updating call sites accordingly)

To revert any given call site to the old behaviour:

http.client.HTTPSConnection(context=ssl._create_unverified_context())
urllib.request.urlopen(context=ssl._create_unverified_context())
urllib.request.URLOpener(context=ssl._create_unverified_context())
urllib.request.FancyURLOpener(context=ssl._create_unverified_context())

And to revert to the old default behaviour globally:

import ssl
ssl._create_https_context = ssl._create_unverified_context

The backport to 2.7 would then be a matter of bringing urllib,
urllib2, httplib and ssl into line with their 3.4.2 counterparts.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

2014-09-09 Thread Christian Heimes 
On 09.09.2014 05:03, Nick Coghlan wrote:
> 
> On 9 Sep 2014 10:48, "Jim J. Jewett"  > wrote:
>> I assume that adding _unverified_urlopen or urlopen(context=...) do
>> provide incremental improvements compatible with the eventual full
>> opt-in.  If so, adding them is probably reasonable, but I think the
>> PEP should explicitly list all such approved half-measures as a guard
>> against API feature creep.
> 
> From Guido's and your feedback, I think we may need two things to
> approve this for 3.4.2 (putting 2.7 aside for now):
> 
> 1. "context" parameter support in urllib.request (to opt out on a
> per-call basis)
> 2. a documented way to restore the old behaviour via sitecustomize
> (which may involve monkeypatching)

What's with our plan to introduce sslcustomize? Is the idea for a
configuration module and named contexts off the table?


For reference:

I came up with the idea to introduce the module "sslcustomize" similar
to sitecustomize. Contrary to sitecustomize the module is imported at
the end of the ssl module.

Based on my idea Nick proposed another addition to the SSL module. He
proposed a ssl.named_contexts mapping from module names to factory
functions that create SSLContext objects.
http://permalink.gmane.org/gmane.comp.python.devel/149292

I still prefer the general idea over the monkey patching idea because it
provides a clean but simple interface for structured configuration.
Monkey patching of stdlib modules is ugly and error-prone.

Christian
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Suggested changes to verify HTTPS by default (was Re: Proposed schedule for 3.4.2)

2014-09-09 Thread Guido van Rossum 
I'm going to leave the design up to Nick and friends for a while. Let me
know when there is a patch to review.

On Tue, Sep 9, 2014 at 3:52 AM, Nick Coghlan  wrote:

> On 9 September 2014 03:44, Alex Gaynor  wrote:
> > *Shifts uncomfortably* it looks like presently there's not a good way to
> > change anything about the SSL configuration for urllib.request.urlopen.
> It
> > does not take a `context` argument, as the http.client API does:
> >
> https://docs.python.org/3/library/urllib.request.html#module-urllib.request
> > and instead takes the cafile, capath, cadefault args.
> >
> > This would need to be updated first, once it *did* take such an argument,
> > this would be accomplished by:
> >
> > context = ssl.create_default_context()
> > context.verify_mode = CERT_OPTIONACERT_NONE
> > context.verify_hostname = False
> > urllib.request.urlopen("
> https://something-i-apparently-dont-care-much-about";,
> > context=context)
>
> I'd never needed to use the existing global configuration settings in
> urllib.request before, but it actually *does* already support setting
> the default opener for urllib.urlopen.
>
> To explicitly set it to use verified HTTPS by default:
>
> import ssl, urllib.request
> https_handler = HTTPSHandler(context=ssl.create_default_context(),
> check_hostname=True)
>
> urllib.request.install_opener(urllib.request.build_opener(https_handler)
>
> When the default changes, turning off verification by default for
> urllib.request.urlopen would look like:
>
> import ssl, urllib.request
> unverified_context = ssl.create_default_context()
> unverified_context.verify_mode = CERT_OPTIONACERT_NONE
> unverified_context.verify_hostname = False
> unverified_handler = HTTPSHandler(context=unverified_context,
> check_hostname=False)
>
> urllib.request.install_opener(urllib.request.build_opener(unverified_handler)
>
> However, even setting the opener like that still leaves
> http.client.HTTPSConnection, urllib.request.URLOpener and
> urllib.request.FancyURLOpener using unverified HTTPS with no easy way
> to change their default behaviour.
>
> That means some other approach to global configuration is going to be
> needed to cover the "coping with legacy corporate infrastructure"
> case, and I still think a monkeypatching based hack is likely to be
> our best bet.
>
> So, focusing on 3.4, but in a way that should be feasible to backport,
> the changes that I now believe would be needed are:
>
> 1. Add "context" arguments to urlopen, URLOpener and FancyURLOpener
> (the latter two have been deprecated since 3.3, but this would make
> things easier for a subsequent 2.7 backport)
> 2. Add a ssl._create_https_context() alias for ssl.create_default_context()
> 3. Change urllib.request.urlopen() and http.client.HTTPSConnection to
> call ssl_create_https_context() rather than
> ssl._create_stdlib_context()
> 4. Rename ssl._create_stdlib_context() to
> ssl._create_unverified_context() (updating call sites accordingly)
>
> To revert any given call site to the old behaviour:
>
> http.client.HTTPSConnection(context=ssl._create_unverified_context())
> urllib.request.urlopen(context=ssl._create_unverified_context())
> urllib.request.URLOpener(context=ssl._create_unverified_context())
> urllib.request.FancyURLOpener(context=ssl._create_unverified_context())
>
> And to revert to the old default behaviour globally:
>
> import ssl
> ssl._create_https_context = ssl._create_unverified_context
>
> The backport to 2.7 would then be a matter of bringing urllib,
> urllib2, httplib and ssl into line with their 3.4.2 counterparts.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   [email protected]   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

2014-09-09 Thread Nick Coghlan 
On 10 Sep 2014 02:11, "Christian Heimes"  wrote:
>
> On 09.09.2014 05:03, Nick Coghlan wrote:
> >
> > On 9 Sep 2014 10:48, "Jim J. Jewett"  > > wrote:
> >> I assume that adding _unverified_urlopen or urlopen(context=...) do
> >> provide incremental improvements compatible with the eventual full
> >> opt-in.  If so, adding them is probably reasonable, but I think the
> >> PEP should explicitly list all such approved half-measures as a guard
> >> against API feature creep.
> >
> > From Guido's and your feedback, I think we may need two things to
> > approve this for 3.4.2 (putting 2.7 aside for now):
> >
> > 1. "context" parameter support in urllib.request (to opt out on a
> > per-call basis)
> > 2. a documented way to restore the old behaviour via sitecustomize
> > (which may involve monkeypatching)
>
> What's with our plan to introduce sslcustomize? Is the idea for a
> configuration module and named contexts off the table?

I'd still like to do that for 3.5+ as the proper long term fix, but I don't
think it's necessary for 2.7 or 3.4.

Cheers,
Nick.

>
>
> For reference:
>
> I came up with the idea to introduce the module "sslcustomize" similar
> to sitecustomize. Contrary to sitecustomize the module is imported at
> the end of the ssl module.
>
> Based on my idea Nick proposed another addition to the SSL module. He
> proposed a ssl.named_contexts mapping from module names to factory
> functions that create SSLContext objects.
> http://permalink.gmane.org/gmane.comp.python.devel/149292
>
> I still prefer the general idea over the monkey patching idea because it
> provides a clean but simple interface for structured configuration.
> Monkey patching of stdlib modules is ugly and error-prone.
>
> Christian
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

2014-09-09 Thread Jim J. Jewett 
On Tue, Sep 9, 2014 at 12:11 PM, Christian Heimes  wrote:
> On 09.09.2014 05:03, Nick Coghlan wrote:
>>
>> On 9 Sep 2014 10:48, "Jim J. Jewett" > > wrote:
>> From Guido's and your feedback, I think we may need two things to
>> approve this for 3.4.2 (putting 2.7 aside for now):

>> 1. "context" parameter support in urllib.request (to opt out on a
>> per-call basis)
>> 2. a documented way to restore the old behaviour via sitecustomize
>> (which may involve monkeypatching)

> What's with our plan to introduce sslcustomize? Is the idea for a
> configuration module and named contexts off the table?

In a perfect world, half-measures would not be needed, and so neither
would sslcustomize.

In the real world, half-measures are needed, but offering too many of
them adds so much confusion that things can actually get worse in
practice.

In other words, sslcustomize could be great, but getting it wrong
would be a step backwards -- so start it as a 3rd party module.  Since
the biggest users are likely supported customers of downstream
distributions, it makes sense to let them take the lead, though I'm
sure they would appreciate a proof of concept.

> I still prefer the general idea over the monkey patching idea because it
> provides a clean but simple interface for structured configuration.
> Monkey patching of stdlib modules is ugly and error-prone.

The primary use case for monkey patching is to support Separation of
Roles.  (Exact titles will of course differ by business.)

If you need structured configuration, then you are already treating
some calls differently from others, which means that you are already
doing partial remediation.  I agree that monkey patching is the wrong
choice if you are doing partial remediation.

But this partial remediation also implies that a "Developer" and
"Business Owner" are involved to decide which calls need to be
changed, and whether to change the call vs dropping the functionality
vs convincing the owner of the other end of the connection to do
things right in the first place.

A Developer in charge of her own environment doesn't need to monkey
patch -- but she could just do the right thing today, or switch to a
framework that does. sslcustomize may be a really good way for her to
document "these are the strange exceptions in our existing
environment", if it is done right.

A "Deployment Engineer" may not even know python, and is certainly not
authorized to make changes beyond "configuration".  Convincing someone
that a .py file is a configuration knob probably requires an exception
that is painful to get.  (And saying "oh, this is just where we list
security stuff that we're ignoring" won't make it easier.)  Changing
the the urlopen calls would therefore be unacceptable even if source
code were available -- and sometimes it isn't.

The Deployment Engineer is often responsible for upgrading the
infrastructure components (possibly including python) for security
patches, so he has to be able to deploy 3.4.x or 2.7.y (though
*probably* not 3.5.0) without any changes to the application itself --
and usually without access to whatever regression tests the
application itself uses.  (Ideally, someone else who does have that
access is involved, but ... not always.)

What the Deployment Engineer *can* do is modify the environment around
the application.  He can write a shell script that sets environment
variables and or command line options.  He can probably even add a
required component -- which might in practice just be a pre-written
module like sslcustomize, or a .pth file that does the monkey patch on
each launch.  But *adding* such a component is orders of magnitude
simpler (from a bureaucratic perspective) than *modifying* one that
already exists.

The end user often can't do anything outside the application's own UI,
which is why the change has to be handled once at deployment, instead
of as break-fix per call site or per bad certificate.

-jJ
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Backwards compatibility after certificate autovalidation

2014-09-09 Thread Nick Coghlan 
On 10 September 2014 07:13, Jim J. Jewett  wrote:
> On Tue, Sep 9, 2014 at 12:11 PM, Christian Heimes  
> wrote:
>> On 09.09.2014 05:03, Nick Coghlan wrote:
>>>
>>> On 9 Sep 2014 10:48, "Jim J. Jewett" >> > wrote:
>>> From Guido's and your feedback, I think we may need two things to
>>> approve this for 3.4.2 (putting 2.7 aside for now):
>
>>> 1. "context" parameter support in urllib.request (to opt out on a
>>> per-call basis)
>>> 2. a documented way to restore the old behaviour via sitecustomize
>>> (which may involve monkeypatching)
>
>> What's with our plan to introduce sslcustomize? Is the idea for a
>> configuration module and named contexts off the table?
>
> In a perfect world, half-measures would not be needed, and so neither
> would sslcustomize.
>
> In the real world, half-measures are needed, but offering too many of
> them adds so much confusion that things can actually get worse in
> practice.
>
> In other words, sslcustomize could be great, but getting it wrong
> would be a step backwards -- so start it as a 3rd party module.  Since
> the biggest users are likely supported customers of downstream
> distributions, it makes sense to let them take the lead, though I'm
> sure they would appreciate a proof of concept.

I see the use case for sslcustomize as being different: it wouldn't be
about turning the stdlib SSL security settings *down*, it would be
about selectively turning them *up*.

As PEP 476 says, turning cert validation on for protocols like SMTP is
currently going to cause significantly more problems than turning it
on for HTTPS. The cleaner, more comprehensive sslcustomize model would
provide a way to opt-in on a virtualenv and installation wide basis to
those higher security settings. The fact it would also offer a
slightly cleaner alternative to the HTTPS downgrade monkeypatch would
be an incidental side effect.

The change would touch many more modules than the current proposal
though, which is why I now believe it's better tackled as a 3.5+ only
feature.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com