[Python-Dev] Python 2.6.3
I had previously wanted to release Python 2.6.3 over the summer, but for various personal reasons, the summer was just too insane. I'd like to reschedule a 2.6.3 release, shooting for final release on 25- September. We should probably do a release candidate, so I'd like to make that on 23-September. Does anybody have objections to that schedule? If not, I'll try to spend some time over the next few days looking at outstanding bugs, and marking release blockers, etc. -Barry PGP.sig Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Barry Warsaw wrote: > I had previously wanted to release Python 2.6.3 over the summer, but for > various personal reasons, the summer was just too insane. I'd like to > reschedule a 2.6.3 release, shooting for final release on 25-September. > > We should probably do a release candidate, so I'd like to make that on > 23-September. > > Does anybody have objections to that schedule? If not, I'll try to > spend some time over the next few days looking at outstanding bugs, and > marking release blockers, etc. 2 days seems a little short (particularly allowing 24 hours or so for the Windows and Mac installers to be produced). Haven't we historically left a week between the RC and actual release for maintenance releases? Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
On Sep 9, 2009, at 9:29 AM, Nick Coghlan wrote: Barry Warsaw wrote: I had previously wanted to release Python 2.6.3 over the summer, but for various personal reasons, the summer was just too insane. I'd like to reschedule a 2.6.3 release, shooting for final release on 25- September. We should probably do a release candidate, so I'd like to make that on 23-September. Does anybody have objections to that schedule? If not, I'll try to spend some time over the next few days looking at outstanding bugs, and marking release blockers, etc. 2 days seems a little short (particularly allowing 24 hours or so for the Windows and Mac installers to be produced). Haven't we historically left a week between the RC and actual release for maintenance releases? Actually, I've rarely done rc's for point releases. JFDI :) I still want to release by the 25th, but I'd be willing to move the rc to Monday the 21st. We're really just trying to avoid a brown bag moment, so that should give us enough time to double check the releases. -Barry PGP.sig Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3145
On Tue, Sep 8, 2009 at 19:56, Benjamin Peterson wrote: > 2009/9/7 Eric Pruitt : >> Hello all, >> >> I have been working on adding asynchronous I/O to the Python >> subprocess module as part of my Google Summer of Code project. Now >> that I have finished documenting and pruning the code, I present PEP >> 3145 for its inclusion into the Python core code. Any and all feedback >> on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated. > > Hi Eric, > One of the reasons you're not getting many response is that you've not > pasted the contents of the PEP in this message. That makes it really > easy for people to comment on various sections. > > BTW, it seems like you were trying to use reST formatting with the > text PEP layout. Double backquotes only mean something in reST. That reST markup is my fault, not Eric. So used to reST PEPs that I just instinctively added it when I check the PEP in. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
On Sat, Jul 25, 2009 at 19:25, Gregory P. Smith wrote: > On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters wrote: > > > > So attached (and at http://codereview.appspot.com/96125/show ) is a > > preliminary fix, correcting the problem with os.fork(), os.forkpty() and > > os.fork1(). This doesn't expose a general API for C code to use, for two > > reasons: it's not easy, and I need this fix more than I need the API > change > > :-) (I actually need this fix myself for Python 2.4, but it applies > fairly > > cleanly.) > > This looks good to me. > Anyone else want to take a look at this before I check it in? I updated the patch (in Rietveld) to contain some documentation about the hazards of mixing fork and threads, which is the best we can do at the moment, at least without seriously overhauling the threading APIs (which, granted, is not that bad an idea, considering the mess they're in.) I've now thoroughly tested the patch, and for most platforms it's strictly better. On AIX it *may* behave differently (possibly 'incorrectly' for specific cases) if something other than os.fork() calls the C fork() and calls PyOS_AfterFork(), since on AIX it used to nuke the thread lock. *I* think the new behaviour (not nuking the lock) is the correct thing to do, but since most places that release the import lock don't bother to check if the lock was even held, the old behaviour may have been succesfully masking the bug on AIX systems. Perhaps for the backport to 2.6 (which I think is in order, and also in accordance with the guidelines) I should leave the AIX workaround in? Anyone think it should not be removed from 3.x/2.7? > Your idea of making this an API called a 'fork lock' or something > sounds good (though I think it needs a better name. PyBeforeFork & > PyAfterFork?). The subprocess module, for example, disables garbage > collection before forking and restores it afterwards to avoid > http://bugs.python.org/issue1336. That type of thing could also be > done in such a function. > Unfortunately it's rather hard to make those functions work correctly with the current API -- we can't provide functions you can just use as arguments to pthread_atfork because the global interpreter lock is not re-entrant and we have no way of testing whether the current thread holds the GIL. I also get the creepy-crawlies when I look at the various thread_* implementations and see the horribly unsafe things they do (and also, for instance, the PendingCall stuff in ceval.c :S) Unfortunately there's no good way to fix these things without breaking API compatibility, let alone ABI compatibility. > Related to the above subprocess fork + gc bug.. It'd be nice for > CPython to have code that does the fork+misc twiddling+exec all in one > C call without needing to execute Python code in the child process > prior to the exec(). Most of the inner body of subprocess's > _execute_child() method could be done that way. (with the obvious > exception of the preexec_fn) > > > > > To fix the mutex-across-fork problem correctly we should really acquire > > three locks (the import lock, the GIL and the TLS lock, in that order.) > The > > import lock is re-entrant, and the TLS lock is tightly confined to the > > thread-local-storage lookup functions, but the GIL is neither re-entrant > nor > > inspectable. That means we can't use pthread_atfork (we can't tell > whether > > we have the GIL already or not, right before the fork), nor can we > provide a > > convenient API for extension modules to use, really. The best we can do > is > > provide three functions, pthread_atfork-style: a 'prepare' function, an > > 'after-fork-in-child' function, and an 'after-fork-in-parent' function. > The > > 'prepare' function would start by releasing the GIL, then acquire the > import > > lock, the GIL and the TLS lock in that order. It would also record > > *somewhere* the thread_ident of the current thread. The 'in-parent' > function > > would simply release the TLS lock and the import lock, and the 'in-child' > > would release those locks, call the threading._at_fork() function, and > fix > > up the TLS data, using the recorded thread ident to do lookups. The > > 'in-child' function would replace the current PyOS_AfterFork() function > > (which blindly reinitializes the GIL and the TLS lock, and calls > > threading._at_fork().) > > > > Alternatively we could do what we do now, which is to ignore the fact > that > > thread idents may change by fork() and thus that thread-local data may > > disappear, in which case the 'in-child' function could do a little less > > work. I'm suitably scared and disgusted of the TLS implementation, the > > threading implementations we support (including the pthread one) and the > way > > we blindly type-pun a pthread_t to a long, that I'm not sure I want to > try > > and build something correct and reliable on top of it. > > > > -- > > Thomas Wouters > > > > Hi! I'm a .signature virus! copy me into your .signature file to help me
Re: [Python-Dev] Python 2.6.3
In article <[email protected]>, Barry Warsaw wrote: > I still want to release by the 25th, but I'd be willing to move the rc > to Monday the 21st. We're really just trying to avoid a brown bag > moment, so that should give us enough time to double check the releases. The recent release of OS X 10.6 (Snow Leopard) has triggered a fair amount of 2.6 bug tracker activity, since 10.6 now includes 2.6 (2.6.1) and a 64-bit version at that. A number of patches have either just been checked-in over the past couple of weeks or are getting some exposure before check-in. Given the timing and the (appropriate) infrequency of 2.6.x releases, I think it would be unfortunate to push 2.6.3 out the door without ensuring that it works well on 10.6. Therefore, I propose that 2.6.3 should have 10.6 compatibility as a "release goal". Without trying to put Ronald on the spot (too much!), it would be a good idea to get his assessment where things stand wrt 2.6 on 10.6 before setting a final release date. -- Ned Deily, [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
On Wed, Sep 9, 2009 at 9:07 AM, Thomas Wouters wrote: > > > On Sat, Jul 25, 2009 at 19:25, Gregory P. Smith wrote: >> >> On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters wrote: >> > >> > So attached (and at http://codereview.appspot.com/96125/show ) is a >> > preliminary fix, correcting the problem with os.fork(), os.forkpty() and >> > os.fork1(). This doesn't expose a general API for C code to use, for two >> > reasons: it's not easy, and I need this fix more than I need the API >> > change >> > :-) (I actually need this fix myself for Python 2.4, but it applies >> > fairly >> > cleanly.) >> >> This looks good to me. > > Anyone else want to take a look at this before I check it in? I updated the > patch (in Rietveld) to contain some documentation about the hazards of > mixing fork and threads, which is the best we can do at the moment, at least > without seriously overhauling the threading APIs (which, granted, is not > that bad an idea, considering the mess they're in.) I've now thoroughly > tested the patch, and for most platforms it's strictly better. On AIX it > *may* behave differently (possibly 'incorrectly' for specific cases) if > something other than os.fork() calls the C fork() and calls > PyOS_AfterFork(), since on AIX it used to nuke the thread lock. *I* think > the new behaviour (not nuking the lock) is the correct thing to do, but > since most places that release the import lock don't bother to check if the > lock was even held, the old behaviour may have been succesfully masking the > bug on AIX systems. > Perhaps for the backport to 2.6 (which I think is in order, and also in > accordance with the guidelines) I should leave the AIX workaround in? Anyone > think it should not be removed from 3.x/2.7? > >> >> Your idea of making this an API called a 'fork lock' or something >> sounds good (though I think it needs a better name. PyBeforeFork & >> PyAfterFork?). The subprocess module, for example, disables garbage >> collection before forking and restores it afterwards to avoid >> http://bugs.python.org/issue1336. That type of thing could also be >> done in such a function. > > Unfortunately it's rather hard to make those functions work correctly with > the current API -- we can't provide functions you can just use as arguments > to pthread_atfork because the global interpreter lock is not re-entrant and > we have no way of testing whether the current thread holds the GIL. I also > get the creepy-crawlies when I look at the various thread_* implementations > and see the horribly unsafe things they do (and also, for instance, the > PendingCall stuff in ceval.c :S) Unfortunately there's no good way to fix > these things without breaking API compatibility, let alone ABI > compatibility. Take a look at http://code.google.com/p/python-atfork/ which I created to address the general fork+threading with locks held causing deadlocks issue with many standard library modules. Currently it only patches the logging module but I intend to support others. I want to get an atfork mechanism into 2.7/3.2 along with every lock in the standard library making proper use of it. See also http://bugs.python.org/issue6721 I make no attempt to deal with C-level locks, only those acquired from python. It doesn't use pthread_atfork but instead models its behavior after that and wraps os.fork and os.forkpty so that they call the registered atfork methods as appropriate. > >> >> Related to the above subprocess fork + gc bug.. It'd be nice for >> CPython to have code that does the fork+misc twiddling+exec all in one >> C call without needing to execute Python code in the child process >> prior to the exec(). Most of the inner body of subprocess's >> _execute_child() method could be done that way. (with the obvious >> exception of the preexec_fn) >> >> > >> > To fix the mutex-across-fork problem correctly we should really acquire >> > three locks (the import lock, the GIL and the TLS lock, in that order.) >> > The >> > import lock is re-entrant, and the TLS lock is tightly confined to the >> > thread-local-storage lookup functions, but the GIL is neither re-entrant >> > nor >> > inspectable. That means we can't use pthread_atfork (we can't tell >> > whether >> > we have the GIL already or not, right before the fork), nor can we >> > provide a >> > convenient API for extension modules to use, really. The best we can do >> > is >> > provide three functions, pthread_atfork-style: a 'prepare' function, an >> > 'after-fork-in-child' function, and an 'after-fork-in-parent' function. >> > The >> > 'prepare' function would start by releasing the GIL, then acquire the >> > import >> > lock, the GIL and the TLS lock in that order. It would also record >> > *somewhere* the thread_ident of the current thread. The 'in-parent' >> > function >> > would simply release the TLS lock and the import lock, and the >> > 'in-child' >> > would release those locks, call the threading._at_fork() function, and >> > fix >> > up the TLS data
Re: [Python-Dev] pthreads, fork, import, and execvp
On Wed, Sep 9, 2009 at 11:36 AM, Thomas Wouters wrote: > > On Wed, Sep 9, 2009 at 20:19, Gregory P. Smith wrote: >> >> Take a look at http://code.google.com/p/python-atfork/ which I created >> to address the general fork+threading with locks held causing >> deadlocks issue with many standard library modules. Currently it only >> patches the logging module but I intend to support others. I want to >> get an atfork mechanism into 2.7/3.2 along with every lock in the >> standard library making proper use of it. >> >> See also http://bugs.python.org/issue6721 >> >> I make no attempt to deal with C-level locks, only those acquired from >> python. It doesn't use pthread_atfork but instead models its behavior >> after that and wraps os.fork and os.forkpty so that they call the >> registered atfork methods as appropriate. > > Well, sure, the *Python code* side of the problem is fixable, thanks to > Python being so awesome ;-P I'm strictly thinking of applications embedding > Python (or even extending it and calling into code that doesn't consider > Python.) Your python-atfork project looks like a terribly good idea, but it > won't fix the embedding/extending issues (nor is it intended to, right?) yeah I decided to only fix what could obviously be fixed and was causing me pain at the moment. ;) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
On Wed, Sep 9, 2009 at 20:19, Gregory P. Smith wrote: > On Wed, Sep 9, 2009 at 9:07 AM, Thomas Wouters wrote: > > > > > > On Sat, Jul 25, 2009 at 19:25, Gregory P. Smith wrote: > >> > >> On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters > wrote: > >> > > >> > So attached (and at http://codereview.appspot.com/96125/show ) is a > >> > preliminary fix, correcting the problem with os.fork(), os.forkpty() > and > >> > os.fork1(). This doesn't expose a general API for C code to use, for > two > >> > reasons: it's not easy, and I need this fix more than I need the API > >> > change > >> > :-) (I actually need this fix myself for Python 2.4, but it applies > >> > fairly > >> > cleanly.) > >> > >> This looks good to me. > > > > Anyone else want to take a look at this before I check it in? I updated > the > > patch (in Rietveld) to contain some documentation about the hazards of > > mixing fork and threads, which is the best we can do at the moment, at > least > > without seriously overhauling the threading APIs (which, granted, is not > > that bad an idea, considering the mess they're in.) I've now thoroughly > > tested the patch, and for most platforms it's strictly better. On AIX it > > *may* behave differently (possibly 'incorrectly' for specific cases) if > > something other than os.fork() calls the C fork() and calls > > PyOS_AfterFork(), since on AIX it used to nuke the thread lock. *I* think > > the new behaviour (not nuking the lock) is the correct thing to do, but > > since most places that release the import lock don't bother to check if > the > > lock was even held, the old behaviour may have been succesfully masking > the > > bug on AIX systems. > > Perhaps for the backport to 2.6 (which I think is in order, and also in > > accordance with the guidelines) I should leave the AIX workaround in? > Anyone > > think it should not be removed from 3.x/2.7? > > > >> > >> Your idea of making this an API called a 'fork lock' or something > >> sounds good (though I think it needs a better name. PyBeforeFork & > >> PyAfterFork?). The subprocess module, for example, disables garbage > >> collection before forking and restores it afterwards to avoid > >> http://bugs.python.org/issue1336. That type of thing could also be > >> done in such a function. > > > > Unfortunately it's rather hard to make those functions work correctly > with > > the current API -- we can't provide functions you can just use as > arguments > > to pthread_atfork because the global interpreter lock is not re-entrant > and > > we have no way of testing whether the current thread holds the GIL. I > also > > get the creepy-crawlies when I look at the various thread_* > implementations > > and see the horribly unsafe things they do (and also, for instance, the > > PendingCall stuff in ceval.c :S) Unfortunately there's no good way to fix > > these things without breaking API compatibility, let alone ABI > > compatibility. > > Take a look at http://code.google.com/p/python-atfork/ which I created > to address the general fork+threading with locks held causing > deadlocks issue with many standard library modules. Currently it only > patches the logging module but I intend to support others. I want to > get an atfork mechanism into 2.7/3.2 along with every lock in the > standard library making proper use of it. > > See also http://bugs.python.org/issue6721 > > I make no attempt to deal with C-level locks, only those acquired from > python. It doesn't use pthread_atfork but instead models its behavior > after that and wraps os.fork and os.forkpty so that they call the > registered atfork methods as appropriate. Well, sure, the *Python code* side of the problem is fixable, thanks to Python being so awesome ;-P I'm strictly thinking of applications embedding Python (or even extending it and calling into code that doesn't consider Python.) Your python-atfork project looks like a terribly good idea, but it won't fix the embedding/extending issues (nor is it intended to, right?) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
On Sep 9, 2009, at 1:29 PM, Ned Deily wrote: In article <[email protected]>, Barry Warsaw wrote: I still want to release by the 25th, but I'd be willing to move the rc to Monday the 21st. We're really just trying to avoid a brown bag moment, so that should give us enough time to double check the releases. The recent release of OS X 10.6 (Snow Leopard) has triggered a fair amount of 2.6 bug tracker activity, since 10.6 now includes 2.6 (2.6.1) and a 64-bit version at that. A number of patches have either just been checked-in over the past couple of weeks or are getting some exposure before check-in. Given the timing and the (appropriate) infrequency of 2.6.x releases, I think it would be unfortunate to push 2.6.3 out the door without ensuring that it works well on 10.6. Therefore, I propose that 2.6.3 should have 10.6 compatibility as a "release goal". Without trying to put Ronald on the spot (too much!), it would be a good idea to get his assessment where things stand wrt 2.6 on 10.6 before setting a final release date. I'm hoping that Python won't have any issues building and running on 10.6, but I don't have it yet so I can't personally test it out. How would you quantify "works well"? Do you have any thoughts on tests you'd run other than the standard test suite? If 2.6.3 is shown to pass its test suite on 10.5.x, is that good enough? Are the specific bug fixes necessary for 10.6? -Barry PGP.sig Description: This is a digitally signed message part ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
On 9 Sep, 2009, at 19:29, Ned Deily wrote: Without trying to put Ronald on the spot (too much!), it would be a good idea to get his assessment where things stand wrt 2.6 on 10.6 before setting a final release date. MacOS X 10.6 support should be stable now, except for a critical issue with IDLE: opening a new window hangs IDLE (issue 6864). That said, I haven't scanned the issue tracker for more 10.6 related issues. BTW. I'm fine with a sept. 25th release, that should give us enough time to shake out issues with OSX 10.6 support. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Thomas Wouters wrote: > Your idea of making this an API called a 'fork lock' or something > sounds good (though I think it needs a better name. PyBeforeFork & > PyAfterFork?). The subprocess module, for example, disables garbage > collection before forking and restores it afterwards to avoid > http://bugs.python.org/issue1336. That type of thing could also be > done in such a function. > > > Unfortunately it's rather hard to make those functions work correctly > with the current API -- we can't provide functions you can just use as > arguments to pthread_atfork because the global interpreter lock is not > re-entrant and we have no way of testing whether the current thread > holds the GIL. I thought "make sure I have the GIL, either by already having it or waiting for it if I don't yet have it" was the entire point of the PyGILState_Ensure() API? [1] Cheers, Nick. [1] http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
