[Python-Dev] Vilnius/Post EuroPython PyPy Sprint 12-14th of July
Vilnius/Post EuroPython PyPy Sprint 12-14th of July The PyPy team is sprinting at EuroPython again and we invite you to participate in our 3 day long sprint at the conference hotel - Reval Hotel Lietuva. If you plan to attend the sprint we recommend you to listen to the PyPy technical talks (`EuroPython schedule`_) during the conference since it will give you a good overview of the status of development. On the morning of the first sprint day (12th) we will also have a tutorial session for those new to PyPy development. As 3 days is relatively short for a PyPy sprint we suggest to travel back home on the 15th if possible (but it is ok to attend less than 3 days too). -- Goals and topics of the sprint -- There are many possible and interesting sprint topics to work on - here we list some possible task areas: * completing the missing python 2.5 features and support * write or port more extension modules (e.g. zlib is missing) * identify slow areas of PyPy through benchmarking and work on improvements, possibly moving app-level parts of the Python interpreter to interp-level if useful. * there are some parts of PyPy in need of refactoring, we may spend some time on those, for example: - rctypes and the extension compiler need some rethinking - support for LLVM 2.0 for the llvm backend - ... * some JIT improvement work * port the stackless transform to ootypesystem * other interesting stuff that you would like to work on ...;-) Registration If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be sent to that list. Please register by adding yourself on the following list (via svn): http://codespeak.net/svn/pypy/extradoc/sprintinfo/post-ep2007/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint --- Preparation (if you feel it is needed): --- * read the `getting-started`_ pages on http://codespeak.net/pypy * for inspiration, overview and technical status you are welcome to read `the technical reports available and other relevant documentation`_ * please direct any technical and/or development oriented questions to pypy-dev at codespeak.net and any sprint organizing/logistical questions to pypy-sprint at codespeak.net * if you need information about the conference, potential hotels, directions etc we recommend to look at http://www.europython.org. We are looking forward to meet you at the Vilnius Post EuroPython PyPy sprint! The PyPy team .. See also .. .. _getting-started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`the technical reports available and other relevant documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html .. _`EuroPython schedule`: http://indico.cern.ch/conferenceTimeTable.py?confId=13919&showDate=all&showSession=all&detailLevel=contribution&viewMode=room ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Proposal for a new function "open_noinherit" to avoid problems with subprocesses and security risks
I'd like to propose a new function "open_noinherit"
or maybe even a new mode flag "n" for the builtin "open"
(see footnote for the names).
The new function should work exactly like the builtin "open", with one
difference:
The open file is not inherited to any child processes
(whereas files opened with "open" will be inherited).
The new function can be implemented (basically) using
os.O_NOINHERIT on MS Windows
resp. fcntl / FD_CLOEXEC on Posix.
I will post a working Python implementation next week.
There are five reasons for the proposal:
1) The builtin "open" causes unexpected problems in conjunction with
subprocesses,
in particular in multi-threaded programs.
It can cause file permission errors in the subprocess or in the current
process.
On Microsoft Windows, some of the possible file permission errors are
not
documented by Microsoft (thus very few programs written for Windows
will
react properly).
2) Inheriting open file handles to subprocesses is a security risk.
3) For the developer, finding "cause and effect" is *very* hard, in
particular in
multi-threaded programs, when the errors occur only in race-conditions.
4) The problems arise in some of the standard library modules as well,
i.e. shutil.filecopy.
5) Very few developers are aware of the possible problems.
As a work-around, one can replace open with
os.fdopen (os.open (..., + os.O_NOINHERIT), ... )
on Windows, but that's really ugly, hard to read,
may raise a different exception than open (IOError instead of OSError),
and needs careful work to take platform-specific code into account
Here is a single-threaded example to demonstrate the effect:
import os
import subprocess
outf = open ("blah.tmp", "wt")
subprocess.Popen("notepad.exe") # or whatever program you like, but
# It must be a program that does not exit immediately!
# Now the subprocess has inherited the open file handle
# We can still write:
outf.write ("Hello world!\n")
outf.close()
# But we can not rename the file (at least on Windows)
os.rename ("blah.tmp", "blah.txt")
# this fails with OSError: [Errno 13] Permission denied
# Similar problems with other file operations on non-Windows platforms.
Ok, in this little program one can see what is going wrong easily.
But what if the subprocess exits very quickly?
Then perhaps you see the OSError, perhaps not - depending on the process
scheduler
of your operation system.
In a commercial multi-theaded daemon application, the error only occured
under heavy load and was hard to reproduce - and it was even harder to find
the cause.
That's because cause and effect were in two different threads in two
completely different
parts of the program:
- Thread A opens a file and starts to write data
- Thread B starts a subprocess (which inherits the file handle from thread
A!)
- Thread A continues writing to the file and closes it.
- And now it's a race condition:
- a) Thread A wants to rename the file - b) the subprocess exits.
If a) is first: Error, if b) is first: no error.
To make things more complicated, even two subprocesses can disturb each
other.
The new function should be implemented in C ideally, because the GIL could
prevent a thread-switch between os.open and the fcntl.F_SETFD call.
Note that the problem described here arises not only for files, but for
sockets
as well.
See bug 1222790: SimpleXMLRPCServer does not set FD_CLOEXEC
Once there is an easy-to-use, platform-independent, documented builtin
"open_noinherit" (or a new mode flag for "open"), the standard library
should
be considered. For each occurence of "open" or "file", it should be
considered
if it necessary to inherit the file to subprocesses. If not, it should be
replaced
with open_noinherit.
One example is shutil.filecopy, where open_noiherit should be used instead
of open.
The socket module is another candidate, I think - but I'm not sure about
that.
A nice effect of using "open_noinherit" is that - in many cases - one no
longer
needs to speficy close_fds = True when calling subprocess.Popen.
[Note that close_fds is *terribly* slow if MAX_OPEN_FILES is "big", e.g.
800,
see bug 1663329]
Footnote:
While writing this mail, at least 3 times I typed "nonherit" instead of
"noinherit".
So maybe someone can propose a better name?
Or a new mode flag character could be "p" (like "private" or "protected").
Henning
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Weekly Python Patch/Bug Summary
Patch / Bug Summary ___ Patches : 385 open (+21) / 3790 closed (+21) / 4175 total (+42) Bugs: 1029 open (+43) / 6744 closed (+43) / 7773 total (+86) RFE : 262 open ( +4) / 291 closed ( +4) / 553 total ( +8) New / Reopened Patches __ syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 reopened by luke-jr syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 reopened by luke-jr syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 reopened by luke-jr syslog syscall support for SysLogLogger (2007-05-02) http://python.org/sf/1711603 reopened by luke-jr dict size changes during iter (2007-05-24) http://python.org/sf/1724999 opened by Ali Gholami Rudi Line ending bug SimpleXMLRPCServer (2007-05-24) http://python.org/sf/1725295 opened by bgrubbs IDLE - cursor color configuration bug (2007-05-25) http://python.org/sf/1725576 opened by Tal Einat Distutils default exclude doesn't match top level .svn (2007-05-25) http://python.org/sf/1725737 opened by Petteri Räty ftplib.py: IndexError in voidresp occasionally (2007-05-26) http://python.org/sf/1726172 opened by kxroberto Patch to vs 2005 build (2007-05-26) http://python.org/sf/1726195 opened by Joseph Armbruster Windows Build Warnings (2007-05-26) http://python.org/sf/1726196 opened by Joseph Armbruster Line iteration readability (2007-05-26) http://python.org/sf/1726198 opened by Joseph Armbruster SimpleHTTPServer extensions_map (2007-05-26) http://python.org/sf/1726208 opened by Joseph Armbruster ftplib and ProFTPD NLST 226 without 1xx response (2007-05-27) http://python.org/sf/1726451 opened by Kenneth Loafman First steps towards new super (PEP 367) (2007-05-28) CLOSED http://python.org/sf/1727209 opened by Guido van Rossum move intern to sys, make intern() optionally warn (2007-05-31) http://python.org/sf/1728741 opened by Anthony Baxter IDLE - configDialog layout cleanup (2007-06-03) http://python.org/sf/1730217 opened by Tal Einat telnetlib: A callback for monitoring the telnet session (2007-06-04) http://python.org/sf/1730959 opened by Samuel Abels BufReader, TextReader for PEP 3116 "New I/O" (2007-06-04) http://python.org/sf/1731036 opened by Ilguiz Latypov Pruning threading.py from asserts (2007-06-05) CLOSED http://python.org/sf/1731049 opened by Björn Lindqvist Expect skips by platform (2007-06-04) http://python.org/sf/1731169 opened by Matt Kraai Missing Py_DECREF in pysqlite_cache_display (2007-06-05) CLOSED http://python.org/sf/1731330 opened by Tim Delaney Improve doc for time.strptime (2007-06-05) http://python.org/sf/1731659 opened by Björn Lindqvist urllib.urlretrieve/URLopener.retrieve - 'buff' argument (2007-06-05) http://python.org/sf/1731720 opened by Dariusz Suchojad Document the constants in the socket module (2007-06-06) http://python.org/sf/1732367 opened by Björn Lindqvist Allow T_LONGLONG to accepts ints (2007-06-09) CLOSED http://python.org/sf/1733960 opened by Roger Upole _lsprof.c:ptrace_enter_call assumes PyErr_* is clean (2007-06-09) http://python.org/sf/1733973 opened by Eyal Lotem PY_LLONG_MAX and so on (2007-06-09) CLOSED http://python.org/sf/1734014 opened by Hirokazu Yamamoto Fast path for unicodedata.normalize() (2007-06-10) http://python.org/sf/1734234 opened by Rauli Ruohonen patch for bug 1170311 "zipfile UnicodeDecodeError" (2007-06-10) http://python.org/sf/1734346 opened by Alexey Borzenkov platform.py patch to support turbolinux (2007-06-11) CLOSED http://python.org/sf/1734945 opened by Yayati_Turbolinux Fix selectmodule.c compilation on GNU/Hurd (2007-06-11) http://python.org/sf/1735030 opened by Michael Banck Kill StandardError (2007-06-12) CLOSED http://python.org/sf/1735485 opened by Collin Winter asyncore should handle also ECONNABORTED in recv (2007-06-12) http://python.org/sf/1736101 opened by billiejoex asyncore/asynchat patches (2007-06-12) http://python.org/sf/1736190 opened by Josiah Carlson EasyDialogs patch to remove aepack dependency (2007-06-15) http://python.org/sf/1737832 opened by has help() can't find right source file (2007-06-15) http://python.org/sf/1738179 opened by Greg Couch Add a -z interpreter flag to execute a zip file (2007-06-19) http://python.org/sf/1739468 opened by andy-chu zipfile.testzip() using progressive file reads (2007-06-19) http://python.org/sf/1739648 opened by Grzegorz Adam Hankiewicz Patch inspect.py for IronPython / Jython Compatibility (2007-06-19) http://python.org/sf/1739696 opened by Mike Foord Accelerate attr dict lookups (2007-06-19) http:
Re: [Python-Dev] Proposal for a new function "open_noinherit" to avoid problems with subprocesses and security risks
Henning von Bargen schrieb: > I'd like to propose a new function "open_noinherit" > or maybe even a new mode flag "n" for the builtin "open" > (see footnote for the names). Do you have a patch implementing that feature? I believe it's unimplementable in Python 2.x: open() is mapped to fopen(), which does not support O_NOINHERIT. If you don't want the subprocess to inherit handles, why don't you just specify close_fds=True when creating the subprocess? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
