Re: [Python-Dev] Thread/garbage collection race in Popen
On Thu, Oct 4, 2012 at 8:56 PM, Ben Leslie wrote:
> Hi all,
>
> I have a Python program where I have many threads each calling Popen, and
> I was hitting some trouble.
>
> I've been seeing this on 3.2.3, however I believe the same issue is still
> potentially a problem on head.
>
> The error manifests itself when a call to os.close(errpipe_read) fails
> with EBADF (
> http://hg.python.org/releasing/3.2.3/file/86d1421a552c/Lib/subprocess.py#l1314
> )
>
> I believe the root cause of this problem is due to a double close() on a
> different file descriptor (which is then reused as errpipe_read).
>
> The file descriptors: p2cwrite, c2pread and errread are all closed at the
> end of the _execute_child method:
>
>
> http://hg.python.org/releasing/3.2.3/file/86d1421a552c/Lib/subprocess.py#l1351
>
> However, these filedescriptors are wrapped up into file objects during
> __init__ (see:
> http://hg.python.org/releasing/3.2.3/file/86d1421a552c/Lib/subprocess.py#l725
> )
>
> As far as I can tell at the point where the garbage collector kicks in
> Popen.{stdin,stdout,stderr} all go out of scope, and will be deallocated,
> and the underlying filedescriptor closed.
>
> However because the filedescriptor is already closed, and by this time is
> actually reused, this deallocation closes what ends up being an incorrect
> file-descriptor.
>
> Since closing a file object where the underlying fd is already closed is
> silenced (
> http://hg.python.org/releasing/3.2.3/file/86d1421a552c/Modules/_io/iobase.c#l235)
> this would not normally be very apparent.
>
> This race between a new filedescriptor being allocated and the garbage
> collector deallocating the file descriptors certainly hits when using a few
> threads, but I guess depending on the exact behaviour of the garbage
> collector it could potentially also occur in a single threaded case as well.
>
> I think a fix would be to remove the explicit close of these file
> descriptors at the end of _execute_child, and let the garbage collector
> close them. Of course that may leak file descriptors, if the GC doesn't
> kick in for a while, so the other option would be to close the file object,
> rather than just the file descriptor.
>
> Hopefully someone more intimately familiar with the module can point me in
> the right direction to verify this, and provide a fix.
>
> Thanks,
>
> Benno
>
I've filed http://bugs.python.org/issue16140 to track this. In general for
bug reports always start by filing it there (then point to it for
discussion on python-dev). :)
What you describe makes sense to me. I'll follow up there.
-gps (subprocess maintainer)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Split unicodeobject.c into subfiles?
Victor Stinner wrote: > Hi, > > I would like to split the huge unicodeobject.c file into smaller > files. It's just the longest C file of CPython: 14,849 lines. > > I don't know exactly how to split it, but first I would like to know > if you would agree with the idea. > > Example: > - Objects/unicode/codecs.c > - Objects/unicode/mod_format.c > - Objects/unicode/methods.c > - Objects/unicode/operators.c > - etc. > > I don't know if it's better to use a subdirectory, or use a prefix for > new files: Objects/unicode_methods.c, Objects/unicode_codecs.c, etc. > There is already a Python/codecs.c file for example (same filename). Better follow the already existing pattern of using "unicode" as prefix, e.g. unicodectype.c and unicodetype_db.h. > I would like to split the unicodeobject.c because it's hard to > navigate in this huge file between all functions, variables, types, > macros, etc. It's hard to add new code and to fix bugs. For example, > the implementation of str%args takes 1000 lines, 2 types and 10 > functions (since my refactor yesterday, in Python 3.3 the main > function is 500 lines long :-)). > > I only see one argument against such refactoring: it will be harder to > backport/forwardport bugfixes. When making such a change, you have to pay close attention to functions that the compiler can potentially inline. AFAIK, moving such functions into a separate file would prevent such inlining/optimizations, e.g. the str formatter wouldn't be able to inline codec calls if placed in separate .c files. It may be better to split the file into multiple .h files which then get recombined into the one unicodeobject.c file. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 05 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2012-09-27: Released eGenix PyRun 1.1.0 ... http://egenix.com/go35 2012-09-26: Released mxODBC.Connect 2.0.1 ... http://egenix.com/go34 2012-09-25: Released mxODBC 3.2.1 ... http://egenix.com/go33 2012-10-23: Python Meeting Duesseldorf ... 18 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #14446: Remove deprecated tkinter functions: Delete an unused function to
Thank you, Jesus. I've missed this function.
On Fri, Oct 5, 2012 at 5:03 AM, jesus.cea wrote:
> http://hg.python.org/cpython/rev/e278f3ab0190
> changeset: 79484:e278f3ab0190
> user:Jesus Cea
> date:Fri Oct 05 04:02:41 2012 +0200
> summary:
> Issue #14446: Remove deprecated tkinter functions: Delete an unused
> function to avoid a warning
>
> files:
> Modules/_tkinter.c | 30 --
> 1 files changed, 0 insertions(+), 30 deletions(-)
>
>
> diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
> --- a/Modules/_tkinter.c
> +++ b/Modules/_tkinter.c
> @@ -323,36 +323,6 @@
> #endif /* WITH_THREAD */
>
>
> -static char *
> -AsString(PyObject *value, PyObject *tmp)
> -{
> -if (PyBytes_Check(value))
> -return PyBytes_AsString(value);
> -else if (PyUnicode_Check(value)) {
> -PyObject *v = PyUnicode_AsUTF8String(value);
> -if (v == NULL)
> -return NULL;
> -if (PyList_Append(tmp, v) != 0) {
> -Py_DECREF(v);
> -return NULL;
> -}
> -Py_DECREF(v);
> -return PyBytes_AsString(v);
> -}
> -else {
> -PyObject *v = PyObject_Str(value);
> -if (v == NULL)
> -return NULL;
> -if (PyList_Append(tmp, v) != 0) {
> -Py_DECREF(v);
> -return NULL;
> -}
> -Py_DECREF(v);
> -return PyBytes_AsString(v);
> -}
> -}
> -
> -
>
> #define ARGSZ 64
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-checkins
>
--
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should vars() return modifiable dict?
On Wed, Oct 3, 2012 at 11:34 AM, Steven D'Aprano wrote: > I find myself unable to choose between 2) and 4), which suggests that > the status quo wins and we keep the current behaviour. What is the benefit of having a dict that represents a namespace, but whose mutations don't mutate said namespace? Compare with option 2, where the dict is mutable, and whose mutations mutate the namespace it represents. That behavior is altogether significantly less surprising. I've never understood why locals() returned a mutable dictionary either, except that Python has no immutable dictionary type. -- Devin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should vars() return modifiable dict?
On Fri, Oct 5, 2012 at 5:29 PM, Devin Jeanpierre wrote: > On Wed, Oct 3, 2012 at 11:34 AM, Steven D'Aprano wrote: >> I find myself unable to choose between 2) and 4), which suggests that >> the status quo wins and we keep the current behaviour. > > What is the benefit of having a dict that represents a namespace, but > whose mutations don't mutate said namespace? Compare with option 2, > where the dict is mutable, and whose mutations mutate the namespace it > represents. That behavior is altogether significantly less surprising. > > I've never understood why locals() returned a mutable dictionary > either, except that Python has no immutable dictionary type. There's no benefit, it's just a historical artefact imposed by backwards compatibility constraints. We should have taken the opportunity to fix it in Python 3.0 (just as we dropped support for "from x import *" at function scope) but I don't believe anyone suggested it at the time. The problem is that, while updating it to return types.MappingProxyType instead would produce more obvious error messages for those that don't know you can't use it to update local variables inside a function (even though it works at module and class scopes), such a change would *break* existing code which knows the dictionary doesn't get written back, and just uses it for its own purposes (e.g passing it to exec). As for *why* changes don't get written back, it's because the compiler is allowed to assume it knows about every variable name that exists in the local scope (by design), and that doesn't fit with writable locals() for functions. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should vars() return modifiable dict?
On 10/05/2012 01:59 PM, Devin Jeanpierre wrote: I've never understood why locals() returned a mutable dictionary either, except that Python has no immutable dictionary type. Ah, but these days it has types.MappingProxyType which provides a read-only wrapper around another mapping type. Perhaps that should be employed here. //arry/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should vars() return modifiable dict?
On 05/10/12 22:58, Nick Coghlan wrote about locals():
As for *why* changes don't get written back, it's because the compiler
is allowed to assume it knows about every variable name that exists in
the local scope (by design), and that doesn't fit with writable
locals() for functions.
And to be clear, that is implementation-dependent. IronPython locals()
does write back to local variables.
Trying to modify locals() can lead to some really unintuitive behaviour.
For example, in CPython 3.2:
py> def test():
... x = 1
... locals()['x'] = 2 # writing to locals has no effect
... locals()['y'] = 3 # except when it does...
... print(x, locals())
...
py> test()
1 {'y': 3, 'x': 1}
--
Steven
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2012-09-28 - 2012-10-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3770 (+16) closed 24149 (+43) total 27919 (+59) Open issues with patches: 1676 Issues opened (44) == #3982: support .format for bytes http://bugs.python.org/issue3982 reopened by exarkun #15111: Wrong ImportError message with importlib http://bugs.python.org/issue15111 reopened by brett.cannon #15379: Charmap decoding of no-BMP characters http://bugs.python.org/issue15379 reopened by serhiy.storchaka #16086: tp_flags: Undefined behaviour with 32 bits long http://bugs.python.org/issue16086 opened by haypo #16088: http.server's send_error could set the content-length of the e http://bugs.python.org/issue16088 opened by pitrou #16094: Tuple extraction in a lambda isn't supported by 2to3 http://bugs.python.org/issue16094 opened by GlitchMr #16095: urllib2 failing with squid proxy and digest authentication http://bugs.python.org/issue16095 opened by toobaz #16096: Get rid of dangerous integer overflow tricks http://bugs.python.org/issue16096 opened by serhiy.storchaka #16098: Bisect optimization in heapq.nsmallest is never used http://bugs.python.org/issue16098 opened by haldean #16099: robotparser doesn't support request rate and crawl delay param http://bugs.python.org/issue16099 opened by XapaJIaMnu #16100: Compiling vim with Python 3.3 support fails http://bugs.python.org/issue16100 opened by v_core #16101: Verify all imported modules at startup are needed http://bugs.python.org/issue16101 opened by brett.cannon #16102: uuid._netbios_getnode() is outdated http://bugs.python.org/issue16102 opened by serhiy.storchaka #16103: Help() fails at raw_input readline (IDLE 2.7.3, Win7, pythonw) http://bugs.python.org/issue16103 opened by terry.reedy #16104: Use multiprocessing in compileall script http://bugs.python.org/issue16104 opened by dholth #16105: Pass read only FD to signal.set_wakeup_fd http://bugs.python.org/issue16105 opened by felipecruz #16106: antigravity tests http://bugs.python.org/issue16106 opened by ezio.melotti #16107: distutils2.version doesn't str() "1.0.post1" correctly http://bugs.python.org/issue16107 opened by richard #16108: Include maintainer information in register/upload http://bugs.python.org/issue16108 opened by richard #16110: Provide logging.config.configParserConfig http://bugs.python.org/issue16110 opened by thbach #16111: Python 2.7.3 Windows MSI installer installs the VC++ 9 dlls di http://bugs.python.org/issue16111 opened by freaksterrao #16112: platform.architecture does not correctly escape argument to /u http://bugs.python.org/issue16112 opened by David.Benjamin #16113: Add SHA-3 (Keccak) support http://bugs.python.org/issue16113 opened by christian.heimes #16114: incorrect path in subprocess.Popen() FileNotFoundError message http://bugs.python.org/issue16114 opened by chris.jerdonek #16115: test that executable arg to Popen() takes precedence over args http://bugs.python.org/issue16115 opened by chris.jerdonek #16116: Can not install C extension modules to inside a venv on Python http://bugs.python.org/issue16116 opened by msmhrt #16117: python2.7.3 struct misaligned when returned http://bugs.python.org/issue16117 opened by iroli #16119: Python 2.7 _socket DLL import error on Windows Vista http://bugs.python.org/issue16119 opened by sk7 #16120: Use |yield from| in stdlib http://bugs.python.org/issue16120 opened by berker.peksag #16121: shlex.shlex.error_leader() reports incorrect line number http://bugs.python.org/issue16121 opened by Arfrever #16123: IDLE - deprecate running without a subprocess http://bugs.python.org/issue16123 opened by serwy #16124: fcntl_ioctl still not 100% sane with unsigned longs http://bugs.python.org/issue16124 opened by yaneurabeya #16125: open accepts arbitrary mode strings as long as they contain U http://bugs.python.org/issue16125 opened by nneonneo #16128: hashable documentation error http://bugs.python.org/issue16128 opened by max #16129: No good way to set 'PYTHONIOENCODING' when embedding python. http://bugs.python.org/issue16129 opened by ideasman42 #16132: ctypes incorrectly encodes .format attribute of memory views http://bugs.python.org/issue16132 opened by dabeaz #16133: asyncore.dispatcher.recv doesn't handle EAGAIN / EWOULDBLOCK http://bugs.python.org/issue16133 opened by Nidan #16134: Add support for RTMP schemes to urlparse http://bugs.python.org/issue16134 opened by Jorge.Gomes #16136: Removal of VMS support http://bugs.python.org/issue16136 opened by jcea #16137: Using time.asctime() with an array with negative tm_hour cause http://bugs.python.org/issue16137 opened by Viktor.Chynarov #16139: Python 3.3 fails when starting from read-only FS http://bugs.python.org/issue16139 opened by andrewjcg #161
Re: [Python-Dev] Should vars() return modifiable dict?
On 10/5/2012 9:01 AM, Larry Hastings wrote: On 10/05/2012 01:59 PM, Devin Jeanpierre wrote: I've never understood why locals() returned a mutable dictionary either, except that Python has no immutable dictionary type. Ah, but these days it has types.MappingProxyType which provides a read-only wrapper around another mapping type. Perhaps that should be employed here. I would like to put this particular tracker issue in a larger context. I believe that the issue of expanding the meaning of vars() intersects with the issue of changing the return type of stat and other functions from a tuple or elaborated tuple to a read-only attribute namespace object (as I believe Larry and some others wish to do). I understand the proposal to be to change to something like a named tuple, with dotted name access to attributes, but without all the tuple behaviors. The are two major behaviors of tuple to be considered. One is indexibility. That, I agree, can go. The other is iterability, which also means introspect-ability. That, I would argue, should not go. However, one might want to iterate or introspect names, values, or pairs. We already have a way to do that -- dicts. So I propose that the meaning of vars(ob), where ob is an object with attributes, should be expanded from 'return ob.__dict__, if there is one' to 'return a name-attribute dict for ob (which will be ob.__dict__ when there is one)'. I believe this is in line with the original intent of vars(), given that all attribute objects once had a __dict__. (I could be mistaken here, but my memory of 1.4 on this is a bit fuzzy.) The addition of __slots__ broke that, but it was not the only one. Giving a meaning to vars(slotted-object) is a partial step in fixing the break, but only a partial step. One way to expand the meaning of vars is to update it as necessary when new internal types or mechanisms are added. This is the proposal in the tracker. But this is contrary to the principle of information localization and using __special__ names to access type-specific behavior. An alternative to teaching vars about various categories and classes of objects would be to give slotted objects, etcetera, a __dict__ property. Its __get__ could return a dict_proxy or whatever. This would be a better place to localize knowledge of how to get the name-attribute mapping. Then vars() would work on such objects as it currently is, without change. This is similar to how iter() and next() work with new classes -- the classes provide the necessary special method. A solution the to problem of an attribute object not having a __dict__ for vars() to return is to give it one. Coming back to my starting point, if read-only attribute objects have a __dict__ property, they could be introspected and indirectly iterated in three different ways, without having __iter__ and __next__. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Split unicodeobject.c into subfiles?
On Thu, Oct 4, 2012 at 6:49 PM, Stephen J. Turnbull wrote: > Chris Jerdonek writes: > > > You can create multiple files this way. I just verified it. But the > > problem happens with merging. You will create merge conflicts in the > > deleted portions of every split file on every merge. There may be a > > way to avoid this that I don't know about though (i.e. to record that > > merges into the deleted portions should no longer occur). > ... > There's no other way to do it that I know of in any VCS because they > all track conflicts at the file level. (It would be straightforward > to generalize git to handle this gracefully, but it would be a hugely > disruptive change. I don't know if Mercurial would be susceptible to > such an extension.) FWIW, I filed an issue in Mercurial's tracker to add support for splitting files and copying subsets of files: http://bz.selenic.com/show_bug.cgi?id=3649 As I thought it might be, the idea was rejected. --Chris ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bad python 2.5 build on OSX 10.8 mountain lion
In article , Ned Deily wrote: > In article <[email protected]>, > Stefan Krah wrote: > > Ned Deily wrote: > > > > Forgot the link... > > > > http://code.google.com/p/googleappengine/issues/detail?id=7885 > > > > On Monday, October 1, 2012, Guido van Rossum wrote: > > > > > As discussed here, the python 2.5 binary distributed by Apple on > > > > > mountain > > > > > lion is broken. Could someone file an official complaint? > > > I've filed a bug against 10.8 python2.5. The 10.8 versions of Apple's > > > pythons are compile with clang and we did see some sign extension issues > > > with ctypes. The 10.7 version of Apple's python2.5 is compiled with > > > llvm-gcc and handles 2**31 correctly. > > Yes, this looks like http://bugs.python.org/issue11149 . > Ah, right, thanks. I've updated the Apple issue accordingly. Update: the bug I filed has been closed as a duplicate of #11932488 which apparently at the moment is still open. No other information is available. -- Ned Deily, [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
