Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Victor Stinner 
> Library code should not be relying on globals settings that can change.

Did you try my implementation of the PEP 433 on your project? Did you
review my patch implementing the PEP 433?
http://hg.python.org/features/pep-433
http://bugs.python.org/issue17036

I expected more change, whereas only very few modules of the stdlib
need changes to support both modes (cloexec=False or cloexec=True by
default). I made 5 different types of changes. Only th change (3)
gives you an idea of how many modules must be fixed to support
cloexec=True by default: 4 modules (http.server, multiprocessing, pty,
subprocess) on a total of something like 200 modules (stdlib). And I'm
not sure that all these modules need to be modified, I have to check
again.

By the way, I chose to not consider file descriptors 0, 1 and 2 as
special: the developer must disable cloexec explicitly for standard
streams. If we consider them as special, fewer (or no) modules would
require changes.

--

(1) add a cloexec parameter, modified modules:

 * io
 * asyncore
 * socket

(2) explicitly enable cloexec: usually for security reasons, don't
leak a file descriptor in a child process => it's not directly related
to this PEP, I should maybe do this in a different commit. Said
differently: code works with cloexec enabled or disabled, but I
consider that enabled is safer.

 * cgi
 * getpass
 * importlib
 * logging
 * os
 * pty
 * pydoc
 * shutil

(3) explicitly disable cloexec: code doesn't work if cloexec is set
(code relies on file descriptor inherance); modified modules:

 * http.server: CGI uses dup2() to replace stdin and stdout
 * multiprocessing: stdin is replaced with /dev/null
 * pty: dup2() to replace stdin, stdout, stderr
 * subprocess: dup2() to replace stdin, stdout, stderr

(4) refactoring to use the new cloexec parameter:

 * tempfile

(5) apply the default value of the cloexec parameter (in C modules):

 * curses
 * mmap
 * oss
 * select
 * random

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Ralf Schmitt 
Antoine Pitrou  writes:

> Yes, it's fine, because an application developer can often control (or
> at least study) the behaviour of all the code involved.

I'd rather not have to check if some library messes with that global
setting and work around it if it does! The fact that you can control and
work around a global setting that may change, isn't a reason to
introduce that global setting and the increased complexity that comes
with it.

>
> It's the same story as with logging configuration and similar
> process-wide settings. Libraries shouldn't mess with it but the
> top-level application definitely can (and should, even).

That's a bad comparison, because the situation is quite different with
the logging module. Configuration of the logging module does not change
the behaviour for code calling into the logging module (well, besides
log output being produced somewhere, but that doesn't matter for the
caller).
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Antoine Pitrou 
Le Tue, 29 Jan 2013 09:35:40 +0100,
Ralf Schmitt  a écrit :
> Antoine Pitrou  writes:
> 
> > Yes, it's fine, because an application developer can often control
> > (or at least study) the behaviour of all the code involved.
> 
> I'd rather not have to check if some library messes with that global
> setting and work around it if it does!

Then just don't try changing the flag. Problem solved.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Ralf Schmitt 
Antoine Pitrou  writes:

> Le Tue, 29 Jan 2013 09:35:40 +0100,
> Ralf Schmitt  a écrit :
>> 
>> I'd rather not have to check if some library messes with that global
>> setting and work around it if it does!
>
> Then just don't try changing the flag. Problem solved.

I was talking about some library changing the flag. Not changing the
flag myself doesn't help in that case. I'll probably have to explicitly
pass a cloexec argument to each fd creating function I'm calling in
order to not be dependent on the global setting.

Please just acknowledge that having a global configurable setting may
lead to problems. Why is sys.setdefaultencoding being hidden in python
2?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Antoine Pitrou 
Le Tue, 29 Jan 2013 11:24:49 +0100,
Ralf Schmitt  a écrit :
> Please just acknowledge that having a global configurable setting may
> lead to problems.

Ralf, I won't "acknowledge" anything just so that it makes you
feel better. Your point has been made and has also been rebutted.
Let's agree to disagree and move along.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013-01-29 Thread Nick Coghlan 
On Tue, Jan 29, 2013 at 4:13 PM, Charles-François Natali
 wrote:
> So I'm definitely -1 against any form of tunable value (be it a
> sys.setdefaultcloexec(), an environment variable or command-line
> flag), and still against changing the default value.

I now think the conservative option is to initially implement the
design in 3.4 without an option to change the global default. The
"cloexec=True/False" model at least *supports* a tunable default, as
well as eventually *changing* the default if we choose to do so, but I
don't think there's any hurry to proceed to those steps.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 433: second try

2013-01-29 Thread Victor Stinner 
Hi,

Here is a new version of my PEP 433. The default value of the cloexec
parameter is now configurable (cmdline option, env var,
sys.setdefaultcloexec), it can be disabled using
sys.setdefaultcloexec(False).

HTML version:
http://python.org/dev/peps/pep-0433/

Implementation:
http://bugs.python.org/issue17036

My last question is on the implementation: should subprocess.Popen
clears the close-on-exec flag on file descriptors passed to pass_fds?
It makes sense if cloexec is True by default.

--

PEP: 433
Title: Easier suppression of file descriptor inheritance
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 10-January-2013
Python-Version: 3.4


Abstract


Add a new optional *cloexec* parameter on functions creating file
descriptors, add different ways to change default values of this
parameter, and add four new functions:

 * ``os.get_cloexec(fd)``
 * ``os.set_cloexec(fd, cloexec=True)``
 * ``sys.getdefaultcloexec()``
 * ``sys.setdefaultcloexec(cloexec=True)``


Rationale
=

A file descriptor has a close-on-exec flag which indicates if the file
descriptor will be inherited or not.

On UNIX, if the close-on-exec flag is set, the file descriptor is not
inherited: it will be closed at the execution of child processes;
otherwise the file descriptor is inherited by child processes.

On Windows, if the close-on-exec flag is set, the file descriptor is not
inherited; the file descriptor is inherited by child processes if the
close-on-exec flag is cleared and if ``CreateProcess()`` is called with
the *bInheritHandles* parameter set to ``TRUE`` (when
``subprocess.Popen`` is created with ``close_fds=False`` for example).
Windows does now have "close-on-exec" flag but an inherance flag which
is just the opposite value. For example, setting close-on-exec flag set
means clearing the ``HANDLE_FLAG_INHERIT`` flag of an handle.


Status in Python 3.3


On UNIX, the subprocess module closes file descriptors greater than 2 by
default since Python 3.2 [#subprocess_close]_. All file descriptors
created by the parent process are automatically closed in the child
process.

``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of
the listening socket, the parent class ``socketserver.TCPServer``
does not set this flag.

There are other cases creating a subprocess or executing a new program
where file descriptors are not closed: functions of the ``os.spawn*()``
and the ``os.exec*()`` families and third party modules calling
``exec()`` or ``fork()`` + ``exec()``. In this case, file descriptors
are shared between the parent and the child processes which is usually
unexpected and causes various issues.

This PEP proposes to continue the work started with the change in the
subprocess in Python 3.2, to fix the issue in any code, and not just
code using subprocess.


Inherited file descriptors issues
-

Closing the file descriptor in the parent process does not close the
related resource (file, socket, ...) because it is still open in the
child process.

The listening socket of TCPServer is not closed on ``exec()``: the child
process is able to get connection from new clients; if the parent closes
the listening socket and create a new listening socket on the same
address, it would get an "address already is used" error.

Not closing file descriptors can lead to resource exhaustion: even if
the parent closes all files, creating a new file descriptor may fail
with "too many files" because files are still open in the child process.

See also the following issues:

 * `Issue #2320: Race condition in subprocess using stdin
   `_ (2008)
 * `Issue #3006: subprocess.Popen causes socket to remain open after
   close `_ (2008)
 * `Issue #7213: subprocess leaks open file descriptors between Popen
   instances causing hangs `_ (2009)
 * `Issue #12786: subprocess wait() hangs when stdin is closed
   `_ (2011)


Security


Leaking file descriptors is a major security vulnerability. An
untrusted child process can read sensitive data like passwords and
take control of the parent process though leaked file descriptors. It
is for example a known vulnerability to escape from a chroot.

See also the CERT recommandation:
`FIO42-C. Ensure files are properly closed when they are no longer needed
`_.


Example of vulnerabilities:


 * `OpenSSH Security Advisory: portable-keysign-rand-helper.adv
   `_
   (April 2011)
 * `CWE-403: Exposure of File Descriptor to Unintended Control Sphere
   `_ (2008)
 * `Hijacking Ap

Re: [Python-Dev] PEP 433: second try

2013-01-29 Thread Larry Hastings 

On 01/29/2013 09:00 AM, Victor Stinner wrote:

Hi,

Here is a new version of my PEP 433.
[...]
  * ``os.get_cloexec(fd)``
  * ``os.set_cloexec(fd, cloexec=True)``
  * ``sys.getdefaultcloexec()``
  * ``sys.setdefaultcloexec(cloexec=True)``


Passing no judgment on the PEP otherwise, just a single observation: the 
"cloexec" parameter to "sys.setdefaultcloexec" should not have a default.



//arry/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com