Re: [Python-Dev] GIL removal question
On Sat, Aug 13, 2011 at 2:12 AM, Stefan Behnel wrote: > Guido van Rossum, 12.08.2011 23:38: >> >> On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote: >>> >>> I think I understand the background and need for GIL. Without it Python >>> programs would have been cluttered with lock/synchronized statements and >>> C-extensions would be harder to write. >> >> No, sorry, the first half of this is incorrect: with or without the >> GIL *Python* code would need the same amount of fine-grained locking. >> (The part about C extensions is correct.) I am butting in because this >> is a common misunderstanding that really needs to be squashed whenever >> it is aired -- the GIL does *not* help Python code to synchronize. A >> thread-switch can occur between any two bytecode opcodes. Without the >> GIL, atomic operations (e.g. dict lookups that doesn't require >> evaluation of __eq__ or __hash__ implemented in Python) are still >> supposed to be atomic. > > And in this context, it's worth mentioning that even C code can be bitten by > the GIL being temporarily released when calling back into the interpreter. > Only plain C code sequences safely keep the GIL, including many (but not > all) calls to the C-API. And, though mostly off-topic, the worst problem with C code, calling back into Python, and the GIL that I have seen (several times): Suppose you are calling some complex C library that creates threads itself, where those threads may also call back into Python. Here you have to put a block around each Python callback that acquires the GIL before and releases it after, since the new threads (created by C code) start without the GIL acquired. I remember a truly nasty incident where the latter was done, but the main thread did not release the GIL since it was returning directly to Python (which would of course release the GIL every so many opcodes so the callbacks would run). But under certain conditions the block with the acquire-release-GIL code around a Python callback was invoked in the main thread (when a validation problem was detected early), and since the main thread didn't release the GIL around the call into the C code, it hung in a nasty spot. Add many layers of software, and a hard-to-reproduce error condition that triggers this, and you have a problem that's very hard to debug... -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL removal question
On Sat, 13 Aug 2011 09:08:16 -0400 Guido van Rossum wrote: > > And, though mostly off-topic, the worst problem with C code, calling > back into Python, and the GIL that I have seen (several times): > Suppose you are calling some complex C library that creates threads > itself, where those threads may also call back into Python. Here you > have to put a block around each Python callback that acquires the GIL > before and releases it after, since the new threads (created by C > code) start without the GIL acquired. I remember a truly nasty > incident where the latter was done, but the main thread did not > release the GIL since it was returning directly to Python (which would > of course release the GIL every so many opcodes so the callbacks would > run). But under certain conditions the block with the > acquire-release-GIL code around a Python callback was invoked in the > main thread (when a validation problem was detected early), and since > the main thread didn't release the GIL around the call into the C > code, it hung in a nasty spot. Add many layers of software, and a > hard-to-reproduce error condition that triggers this, and you have a > problem that's very hard to debug... These days we have PyGILState_Ensure(): http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure and even dedicated documentation: http://docs.python.org/dev/c-api/init.html#non-python-created-threads ;) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: Mirroring Python repos to Bitbucket
Charles McLaughlin of Atlassian has set up mirrors of the Mercurial repositories hosted on python.org as part of the ongoing infrastructure improvement work. These mirrors will give us a public fail-over repository in the event that hg.python.org goes offline unexpectedly, and also provide features such as RSS feeds of changes for users interested in monitoring the repository passively. Thank you, Charles for setting this up and Atlassian for hosting it! Doug Begin forwarded message: > From: Charles McLaughlin > Date: August 10, 2011 3:43:06 PM EDT > To: Jesse Noller , Doug Hellmann > Subject: Mirroring Python repos to Bitbucket > > Hey, > > You guys expressed some interest in mirroring repos to Bitbucket a > couple weeks ago. I mentioned we mirror a few Python repos here: > > https://bitbucket.org/mirror/ > > But that doesn't cover everything from hg.python.org. So, I wrote a > little script that scrapes the Python HgWeb and mirrors everything to > Bitbucket. Here's the script in case you're curious: > > https://bitbucket.org/cmclaughlin/mirror-python-repos/ > > We're running the script hourly to keep the mirrors up to date. The > mirrored repos live under this URL: > > https://bitbucket.org/python_mirrors > > A few people here have mentioned "python_mirrors" is a strange name. I > can change that if you'd like. I don't have any better ideas though. > Also, anyone can fork my script if they see any room for improvement :) > > Please feel free to forward this email to mailing lists, etc to get the > word out. > > Regards, > Charles ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mirroring Python repos to Bitbucket
On Sat, 13 Aug 2011 19:08:40 -0400 Doug Hellmann wrote: > > Charles McLaughlin of Atlassian has set up mirrors of the Mercurial > repositories hosted on python.org as part of the ongoing infrastructure > improvement work. These mirrors will give us a public fail-over repository in > the event that hg.python.org goes offline unexpectedly, and also provide > features such as RSS feeds of changes for users interested in monitoring the > repository passively. There is already an RSS feed at http://hg.python.org/cpython/rss-log Another possibility is the gmane mirror of python-checkins, which has its own RSS feed: http://rss.gmane.org/gmane.comp.python.cvs Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL removal question
On Sat, Aug 13, 2011 at 8:43 AM, Antoine Pitrou wrote: > On Sat, 13 Aug 2011 09:08:16 -0400 > Guido van Rossum wrote: >> >> And, though mostly off-topic, the worst problem with C code, calling >> back into Python, and the GIL that I have seen (several times): >> Suppose you are calling some complex C library that creates threads >> itself, where those threads may also call back into Python. Here you >> have to put a block around each Python callback that acquires the GIL >> before and releases it after, since the new threads (created by C >> code) start without the GIL acquired. I remember a truly nasty >> incident where the latter was done, but the main thread did not >> release the GIL since it was returning directly to Python (which would >> of course release the GIL every so many opcodes so the callbacks would >> run). But under certain conditions the block with the >> acquire-release-GIL code around a Python callback was invoked in the >> main thread (when a validation problem was detected early), and since >> the main thread didn't release the GIL around the call into the C >> code, it hung in a nasty spot. Add many layers of software, and a >> hard-to-reproduce error condition that triggers this, and you have a >> problem that's very hard to debug... > > These days we have PyGILState_Ensure(): > http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure > > and even dedicated documentation: > http://docs.python.org/dev/c-api/init.html#non-python-created-threads > > ;) That is awesome! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3154 - pickle protocol 4
On Fri, Aug 12, 2011 at 3:58 AM, Antoine Pitrou wrote: > This PEP is an attempt to foster a number of small incremental > improvements in a future pickle protocol version. The PEP process is > used in order to gather as many improvements as possible, because the > introduction of a new protocol version should be a rare occurrence. Thanks. this sounds like a good idea. That's not to say that I have already approved the PEP. :-) But from skimming it I have no objections except that it needs to be fleshed out. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mirroring Python repos to Bitbucket
On Aug 13, 2011, at 7:23 PM, Antoine Pitrou wrote: > On Sat, 13 Aug 2011 19:08:40 -0400 > Doug Hellmann wrote: >> >> Charles McLaughlin of Atlassian has set up mirrors of the Mercurial >> repositories hosted on python.org as part of the ongoing infrastructure >> improvement work. These mirrors will give us a public fail-over repository >> in the event that hg.python.org goes offline unexpectedly, and also provide >> features such as RSS feeds of changes for users interested in monitoring the >> repository passively. > > There is already an RSS feed at http://hg.python.org/cpython/rss-log > Another possibility is the gmane mirror of python-checkins, which has > its own RSS feed: http://rss.gmane.org/gmane.comp.python.cvs Thanks for the tip, I didn't know about either of those. Doug ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL removal question
On Sun, Aug 14, 2011 at 9:26 AM, Guido van Rossum wrote: >> These days we have PyGILState_Ensure(): >> http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure >> >> and even dedicated documentation: >> http://docs.python.org/dev/c-api/init.html#non-python-created-threads >> >> ;) > > That is awesome! Although, if it's possible to arrange it, it's still better to do that once and then use BEGIN/END_ALLOW_THREADS to avoid the overhead of creating and destroying the temporary thread states: http://blog.ccpgames.com/kristjan/2011/06/23/temporary-thread-state-overhead/ Still, it's far, far easier than it used to be to handle the GIL correctly from non-Python created threads. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Mirroring Python repos to Bitbucket
On Sun, Aug 14, 2011 at 9:08 AM, Doug Hellmann wrote: > > Charles McLaughlin of Atlassian has set up mirrors of the Mercurial > repositories hosted on python.org as part of the ongoing infrastructure > improvement work. These mirrors will give us a public fail-over repository in > the event that hg.python.org goes offline unexpectedly, and also provide > features such as RSS feeds of changes for users interested in monitoring the > repository passively. The main advantage of those mirrors to my mind is that it makes it easy for anyone to clone their own copy of the python.org repos without having to upload the whole thing to bitbucket themselves. That makes it easy for people to use a natural Mercurial workflow to develop and collaborate on patches, even for components other than the main CPython repo (e.g. the devguide or the benchmark suite). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Monotonic, not monotonous
On Sun, Aug 14, 2011 at 9:53 AM, antoine.pitrou wrote: > http://hg.python.org/cpython/rev/0273d0734593 > changeset: 71862:0273d0734593 > user: Antoine Pitrou > date: Sun Aug 14 01:51:52 2011 +0200 > summary: > Monotonic, not monotonous > > files: > Lib/test/pickletester.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) I dunno, I reckon systematically testing pickles could get pretty monotonous, too ;) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL removal question
Den 13.08.2011 17:43, skrev Antoine Pitrou: These days we have PyGILState_Ensure(): http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure With the most recent Cython (0.15) we can just do: with gil: to ensure holding the GIL. And similarly from a thread holding the GIL with nogil: to temporarily release it. There are also some OpenMP support in Cython 0.15. OpenMP is much easier than messing around with threads manually (it moves all the hard parts of multithreading to the compiler). Now Cython almost makes it look Pythonic: http://docs.cython.org/src/userguide/parallelism.html Sturla ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
