[Python-Dev] Tracking fixes of security vulnerabilies: we are good!

2017-10-18 Thread Victor Stinner 
Hi,

Since the beginning of the year, I'm working on a tool to track if all
security vulnerabilities are fixed in all Python maintained versions
(versions still accepting security fixes):

   http://python-security.readthedocs.io/vulnerabilities.html

Currently, five branches are maintained: 2.7, 3.4, 3.5, 3.6 and master.

   https://devguide.python.org/#status-of-python-branches

Thanks to Ned Deily and Georg Brandl, Python 3.3 reached its
end-of-life (EOL) last month, after 5 years of good service (as
expected). It reduced the number of maintained branches from six to
five :-) Python 3.3.7 released last months contains the last security
fixes.

The good news is that we got releases last months with fixes for
almost all security vulnerabilities. Only Python 3.4 and Python 3.5
have two known vulnerabilities, but I consider that their severity is
not high hopefully.

"Expat 2.2.3" is not fixed yet in Python 3.4 and 3.5, but I'm not sure
that Python is really affected by fixed Expat vulnerabilities, since
Python uses its own code to generate a secret key for the Expat "hash
secret". Our embedded expat copy is used on Windows and macOS, but not
on Linux.

"update zlib to 1.2.11" was fixed in the Python 3.4 branch, but no
release was made yet. This issue only impacts Windows. Linux and macOS
use the system zlib.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Guido van Rossum 
On Tue, Oct 17, 2017 at 9:40 PM, Nick Coghlan  wrote:

> On 18 October 2017 at 05:55, Yury Selivanov 
> wrote:
>
>> I actually like what you did in
>> https://github.com/gvanrossum/pep550/blob/master/simpler.py, it seems
>> reasonable.  The only thing that I'd change is to remove "set_ctx"
>> from the public API and add "Context.run(callable)".  This makes the
>> API more flexible to potential future changes and amendments.
>>
>
> Yep, with that tweak, I like Guido's suggested API as well.
>

I've added the suggested Context.run() method.

>
> Attempting to explain why I think we want "Context.run(callable)" rather
> "context_vars.set_ctx()" by drawing an analogy to thread local storage:
>
> 1. In C, the compiler & CPU work together to ensure you can't access
> another thread's thread locals.
>

But why is that so important? I wouldn't recommend doing it, but it might
be handy for a debugger to be able to inspect a thread's thread-locals. As
it is, it seems a debugger can only access thread-locals for the thread in
which the debugger itself runs. It has better access to the real locals on
the thread's stack of frames!


> 2. In Python's thread locals API, we do the same thing: you can only get
> access to the running thread's thread locals, not anyone else's
>

But there's no real benefit in this. In C, I could imagine a compiler
optimizing access to thread-locals, but in Python that's moot.


> At the Python API layer, we don't expose the ability to switch explicitly
> to another thread state while remaining within the current function.
> Instead, we only offer two options: starting a new thread, and waiting for
> a thread to finish execution. The lifecycle of the thread local storage is
> then intrinsically linked to the lifecycle of the thread it belongs to.
>

To me this feels more a side-effect of the implementation (perhaps
inherited from C's implementation) than an intentional design.

To be clear, I think it's totally fine for *clients* of the ContextVar API
-- e.g. numpy or decimal -- to assume that their context doesn't change
arbitrarily while they're happily executing in a single frame or calling
stuff they trust not to change the context. (IOW all changes to a
particular ContextVar would be through that ContextVar object, not through
behind-the-scenes manipulation of the thread's current context).

But for *frameworks* (e.g. asyncio or Twisted) I find it simpler to think
about the context in terms of `set_ctx` and `get_ctx`, and I worry that
*hiding* these might block off certain API design patterns that some
framework might want to use -- who knows, maybe Nathaniel (who is fond of
`with`
)
might come up with a context manager to run a block of code in a different
context (perhaps cloned from the current one).


> That intrinsic link makes various aspects of thread local storage easier
> to reason about, since the active thread state can't change in the middle
> of a running function - even if the current thread gets suspended by the
> OS, resuming the function also implies resuming the original thread.
>

I don't feel reasoning would be much impaired. When reasoning about code we
make assumptions that are theoretically unsafe all the time (e.g. "nobody
will move the clock back").


> Including a "contextvars.set_ctx" API would be akin to making
> PyThreadState_Swap a public Python-level API, rather than only exposing
> _thread.start_new_thread the way we do now.
>

It's different for threads, because they are the bedrock of execution, and
nobody is interested in implementing their own threading framework that
doesn't build on this same bedrock.


> One reason we *don't* do that is because it would make thread locals much
> harder to reason about - every function call could have an implicit side
> effect of changing the active thread state, which would mean the thread
> locals at the start of the function could differ from those at the end of
> the function, even if the function itself didn't do anything to change them.
>

Hm. Threads are still hard to reason about, because for everything *but*
thread-locals there is always the possibility that it's being mutated by
another thread... So I don't think we should get our knickers twisted over
thread-local variables.


> Only offering Context.run(callable) provides a similar "the only changes
> to the execution context will be those this function, or a function it
> called, explicitly initiates" protection for context variables, and Guido's
> requested API simplifications make this aspect even easier to reason about:
> after any given function call, you can be certain of being back in the
> context you started in, because we wouldn't expose any Python level API
> that allowed an execution context switch to persist beyond the frame that
> initiated it.
>

And as long as you're not calling something that's a specific framework's
API for messing with the context, that'

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Yury Selivanov 
On Wed, Oct 18, 2017 at 1:06 PM, Guido van Rossum  wrote:
> On Tue, Oct 17, 2017 at 9:40 PM, Nick Coghlan  wrote:
[..]
>> By contrast, "contextvars.set_ctx" would need various wrappers to handle
>> correctly reverting the context change, and would hence be prone to "changed
>> the active context without changing it back" bugs (which can be especially
>> fun when you're dealing with a shared pool of worker threads or processes).
>
>
> So let's have both.

The main reason why I don't like 'set_ctx()' is because it would make
it harder for us to adopt PEP 550-like design later in the future
(*if* we need that.)

PEP 550 is designed in such a way, that 'generator.send()' is the only
thing that can control the actual stack of LCs.  If users can call
'set_ctx' themselves, it means that it's no longer safe for
'generator.send()' to simply pop the topmost LC from the stack.  This
can be worked around, potentially, but the we don't actually need
'set_ctx' in asyncio or in any other async framework.  There is simply
no hard motivation to have it.  That's why I'd like to have just
Context.run(), because it's sufficient, and it doesn't burn the bridge
to PEP 550-like design.

Yury
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Ethan Furman 

On 10/18/2017 10:50 AM, Yury Selivanov wrote:

On Wed, Oct 18, 2017 at 1:06 PM, Guido van Rossum  wrote:

On Tue, Oct 17, 2017 at 9:40 PM, Nick Coghlan  wrote:

[..]

By contrast, "contextvars.set_ctx" would need various wrappers to handle
correctly reverting the context change, and would hence be prone to "changed
the active context without changing it back" bugs (which can be especially
fun when you're dealing with a shared pool of worker threads or processes).



So let's have both.


The main reason why I don't like 'set_ctx()' is because it would make
it harder for us to adopt PEP 550-like design later in the future
(*if* we need that.)

PEP 550 is designed in such a way, that 'generator.send()' is the only
thing that can control the actual stack of LCs.  If users can call
'set_ctx' themselves, it means that it's no longer safe for
'generator.send()' to simply pop the topmost LC from the stack.


I don't see why this is a concern -- Python is a "consenting adults" language.  If users decide to start mucking around 
with advanced behavior and something breaks, well, they own all the pieces! ;)


Unless it's extremely difficult to not seg-fault in such a situation I don't 
think this is a valid argument.

--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Yury Selivanov 
On Wed, Oct 18, 2017 at 2:10 PM, Ethan Furman  wrote:
[..]
> Unless it's extremely difficult to not seg-fault in such a situation I don't
> think this is a valid argument.

Well, you don't think so, but I do, after writing a few
implementations of this PEP family.  It would complicate the design,
but the function isn't even needed, strictly speaking.

Yury
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Guido van Rossum 
On Wed, Oct 18, 2017 at 10:50 AM, Yury Selivanov 
wrote:

> The main reason why I don't like 'set_ctx()' is because it would make
> it harder for us to adopt PEP 550-like design later in the future
> (*if* we need that.)
>
> PEP 550 is designed in such a way, that 'generator.send()' is the only
> thing that can control the actual stack of LCs.  If users can call
> 'set_ctx' themselves, it means that it's no longer safe for
> 'generator.send()' to simply pop the topmost LC from the stack.  This
> can be worked around, potentially, but the we don't actually need
> 'set_ctx' in asyncio or in any other async framework.  There is simply
> no hard motivation to have it.  That's why I'd like to have just
> Context.run(), because it's sufficient, and it doesn't burn the bridge
> to PEP 550-like design.
>

Honestly that stack-popping in send() always felt fragile to me, so I'd be
happy if we didn't need to depend on it.

That said I'm okay with presenting set_ctx() *primarily* as an educational
tool for showing how Context.run() works. We could name it _set_ctx() and
add a similar note as we have for sys._getframe(), basically keeping the
door open for future changes that may render it non-functional without
worries about backward compatibility (and without invoking the notion of
"provisional" API).

There's no problem with get_ctx() right?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Yury Selivanov 
On Wed, Oct 18, 2017 at 2:21 PM, Guido van Rossum  wrote:
> On Wed, Oct 18, 2017 at 10:50 AM, Yury Selivanov 
> wrote:
>>
>> The main reason why I don't like 'set_ctx()' is because it would make
>> it harder for us to adopt PEP 550-like design later in the future
>> (*if* we need that.)
>>
>> PEP 550 is designed in such a way, that 'generator.send()' is the only
>> thing that can control the actual stack of LCs.  If users can call
>> 'set_ctx' themselves, it means that it's no longer safe for
>> 'generator.send()' to simply pop the topmost LC from the stack.  This
>> can be worked around, potentially, but the we don't actually need
>> 'set_ctx' in asyncio or in any other async framework.  There is simply
>> no hard motivation to have it.  That's why I'd like to have just
>> Context.run(), because it's sufficient, and it doesn't burn the bridge
>> to PEP 550-like design.
>
>
> Honestly that stack-popping in send() always felt fragile to me, so I'd be
> happy if we didn't need to depend on it.
>
> That said I'm okay with presenting set_ctx() *primarily* as an educational
> tool for showing how Context.run() works. We could name it _set_ctx() and
> add a similar note as we have for sys._getframe(), basically keeping the
> door open for future changes that may render it non-functional without
> worries about backward compatibility (and without invoking the notion of
> "provisional" API).

'_set_ctx()' + documentation bits work for me.  I also assume that if
you accept the PEP, you do it provisionally, right?  That should make
it possible for us to *slightly* tweak the
implementation/API/semantics in 3.8 if needed.

> There's no problem with get_ctx() right?

Yes, 'get_ctx()' is absolutely fine.  We still need it for async
tasks/callbacks.

Yury
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Guido van Rossum 
Actually after recent debate  I think
this PEP should *not* be provisional.

On Wed, Oct 18, 2017 at 11:45 AM, Yury Selivanov 
wrote:

> On Wed, Oct 18, 2017 at 2:21 PM, Guido van Rossum 
> wrote:
> > On Wed, Oct 18, 2017 at 10:50 AM, Yury Selivanov <
> [email protected]>
> > wrote:
> >>
> >> The main reason why I don't like 'set_ctx()' is because it would make
> >> it harder for us to adopt PEP 550-like design later in the future
> >> (*if* we need that.)
> >>
> >> PEP 550 is designed in such a way, that 'generator.send()' is the only
> >> thing that can control the actual stack of LCs.  If users can call
> >> 'set_ctx' themselves, it means that it's no longer safe for
> >> 'generator.send()' to simply pop the topmost LC from the stack.  This
> >> can be worked around, potentially, but the we don't actually need
> >> 'set_ctx' in asyncio or in any other async framework.  There is simply
> >> no hard motivation to have it.  That's why I'd like to have just
> >> Context.run(), because it's sufficient, and it doesn't burn the bridge
> >> to PEP 550-like design.
> >
> >
> > Honestly that stack-popping in send() always felt fragile to me, so I'd
> be
> > happy if we didn't need to depend on it.
> >
> > That said I'm okay with presenting set_ctx() *primarily* as an
> educational
> > tool for showing how Context.run() works. We could name it _set_ctx() and
> > add a similar note as we have for sys._getframe(), basically keeping the
> > door open for future changes that may render it non-functional without
> > worries about backward compatibility (and without invoking the notion of
> > "provisional" API).
>
> '_set_ctx()' + documentation bits work for me.  I also assume that if
> you accept the PEP, you do it provisionally, right?  That should make
> it possible for us to *slightly* tweak the
> implementation/API/semantics in 3.8 if needed.
>
> > There's no problem with get_ctx() right?
>
> Yes, 'get_ctx()' is absolutely fine.  We still need it for async
> tasks/callbacks.
>
> Yury
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

2017-10-18 Thread Nick Coghlan 
On 19 October 2017 at 04:53, Guido van Rossum  wrote:

> Actually after recent debate  I think
> this PEP should *not* be provisional.
>

+1 from me - "contextvars._set_ctx()" is the only part I think we're really
unsure about in your latest API design, and marking that specific function
as private will cover the fact that its semantics aren't guaranteed yet.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com