Re: [Python-Dev] hg branch gone?
Brett Cannon schrieb: > I just tried to update my 3.0 branch in hg from > http://code.python.org/hg/branches/py3k/ and hg is telling me it's a > 404. Anyone else having trouble? 404 here too. Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose the problem is that the loggerhead app greedily tries to serve all URLs below code.python.org/. Yet another hint at the evil force-Bazaar-upon-them conspiracy . cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
Georg Brandl wrote: Brett Cannon schrieb: I just tried to update my 3.0 branch in hg from http://code.python.org/hg/branches/py3k/ and hg is telling me it's a 404. Anyone else having trouble? 404 here too. Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose the problem is that the loggerhead app greedily tries to serve all URLs below code.python.org/. Yet another hint at the evil force-Bazaar-upon-them conspiracy . I'm not so sure Bazaar isn't it's own victim :) I posted this yesterday about using bzr: >I'd like to try it out, but the instructions on > http://www.python.org/dev/bazaar/ say to get wget > http://code.python.org/snapshots/python-bzr-snapshot.tar.bz2, which is > a 404. Anyone know the actual URL? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Neil Schemenauer wrote: >> Sigurd Torkel Meldgaard <[EMAIL PROTECTED]> wrote: >>> For a student project in a course on virtual machines, we are >>> evaluating the possibility to experiment with removing the GIL >>> from CPython >> >> Hi, >> >> It's great to hear of this kind of project. I think what you want >> to do is difficult but possible. The major compilcation would be >> that extension modules would have to re-written since they all >> assume a reference counting GC. A foreign function interface like >> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it >> does place different demands on extension module authors. > > Michael Foord's comment about the way Ironclad does it suggests a > possible interim step that would allow existing extensions to be used at > the cost of some additional overhead: use free threading in the core, > but have an "extension module lock" that cuts things back to a single > thread whenever non-core code is invoked. Ahh, or do it per object! Convert the core to use a precise GC references, but retain the refcount API as a sort of forced unknown reference. Needs bodging to retain our current handling of cycles in refcounted objects, but it should be doable, and it potentially has a lot less overhead (and is more scalable) than what I've got now with safethread. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Adam Olsen wrote: > On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: >> Neil Schemenauer wrote: >>> Sigurd Torkel Meldgaard <[EMAIL PROTECTED]> wrote: For a student project in a course on virtual machines, we are evaluating the possibility to experiment with removing the GIL from CPython >>> Hi, >>> >>> It's great to hear of this kind of project. I think what you want >>> to do is difficult but possible. The major compilcation would be >>> that extension modules would have to re-written since they all >>> assume a reference counting GC. A foreign function interface like >>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it >>> does place different demands on extension module authors. >> Michael Foord's comment about the way Ironclad does it suggests a >> possible interim step that would allow existing extensions to be used at >> the cost of some additional overhead: use free threading in the core, >> but have an "extension module lock" that cuts things back to a single >> thread whenever non-core code is invoked. > > Ahh, or do it per object! Convert the core to use a precise GC > references, but retain the refcount API as a sort of forced unknown > reference. Needs bodging to retain our current handling of cycles in > refcounted objects, but it should be doable, and it potentially has a > lot less overhead (and is more scalable) than what I've got now with > safethread. The GIL protects more than just refcounting: it is used widely to protect access to shared data structures both inside and outside the core. Until all the extension modules implement their own locks instead of relying on the GIL, some form of global lock needs to remain if you want to preserve compatibility with existing extensions. And since being able to use existing extensions is the main advantage of using CPython rather than just writing a separate free-threaded Python VM, that seems like a fairly important design goal. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My patches
On Fri, Oct 31, 2008 at 7:46 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: >> >> What about having two level of devs ? >> >> + core developers + standard library developers >> >> [cut] > > > So I'd suggest thinking about developer responsibilities more in terms > of areas of expertise rather than "levels" of developers. Those of us > that happen to understand the guts of the compiler or the VM aren't more > competent globally or more trusted than those maintaining the various > modules in the standard library - just interested in different things. Right, I would like to share my experience about this, if it can be helpful. I have focused so far in distutils, which, I believe was not in the top priority of core developers during the last year. (If this is not true please forgive me). Anyway, so I am starting to become quite specialized in this part of Python, and I pushed patches for it in the tracker. The patches I wrote that made it so far took between 4 to 8 months to be applied. I have really simple patches for distutils that are just adding tests, that are waiting for review. I guess these patches will be reviewed in a few months, I am failry confident about that. I know core developers are drowned into more important topics. And this area of Python is being highly discussed to be refactored, maybe outside the stdlib at some point, but there are a *lot* of simple things to do today in there. So basically, if I want to be efficient in distutils I need to become a core developer, and this means (Guido said) I need to start providing patches for the rest of the Python code and eventually (after a few years I guess) maybe become a core developer. Then I'll be able to work in distutils because at that point in the future I'll be trusted. I can't do that ! I don't have the time to become a Python core code expert. But in my everyday work I became a packaging / deploying specialist. So my point is : if I am "trusted" at some point in the work I am doing in distutils, will I have a commit access there ? If so, I 'll continue to focus on this package and to commit patches for it, to try to gain that trust for sure. Otherwise I will need to work in a third-party library to be efficient, and stop working on patches for distutils. And this is, ihmo a bad thing because this sdtlib package need some love. Cheers Tarek ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 2:11 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Adam Olsen wrote: >> On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: >>> Neil Schemenauer wrote: Sigurd Torkel Meldgaard <[EMAIL PROTECTED]> wrote: > For a student project in a course on virtual machines, we are > evaluating the possibility to experiment with removing the GIL > from CPython Hi, It's great to hear of this kind of project. I think what you want to do is difficult but possible. The major compilcation would be that extension modules would have to re-written since they all assume a reference counting GC. A foreign function interface like CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it does place different demands on extension module authors. >>> Michael Foord's comment about the way Ironclad does it suggests a >>> possible interim step that would allow existing extensions to be used at >>> the cost of some additional overhead: use free threading in the core, >>> but have an "extension module lock" that cuts things back to a single >>> thread whenever non-core code is invoked. >> >> Ahh, or do it per object! Convert the core to use a precise GC >> references, but retain the refcount API as a sort of forced unknown >> reference. Needs bodging to retain our current handling of cycles in >> refcounted objects, but it should be doable, and it potentially has a >> lot less overhead (and is more scalable) than what I've got now with >> safethread. > > The GIL protects more than just refcounting: it is used widely to > protect access to shared data structures both inside and outside the > core. Until all the extension modules implement their own locks instead > of relying on the GIL, some form of global lock needs to remain if you > want to preserve compatibility with existing extensions. And since being > able to use existing extensions is the main advantage of using CPython > rather than just writing a separate free-threaded Python VM, that seems > like a fairly important design goal. safethread already handles that, via monitors. The main thread is given a monitor that all old, unshareable extension modules exist in. Once they're updated and marked as shareable they can be used from any monitor in any thread. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
>> Yet another hint at the evil force-Bazaar-upon-them conspiracy . > > I'm not so sure Bazaar isn't it's own victim :) I have now restored the original URL structure, and moved the loggerhead installation to http://code.python.org/loggerhead/ Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 31, 2008, at 08:44 AM, Georg Brandl wrote: >Brett Cannon schrieb: >> I just tried to update my 3.0 branch in hg from >> http://code.python.org/hg/branches/py3k/ and hg is telling me it's a >> 404. Anyone else having trouble? > >404 here too. > >Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose >the problem is that the loggerhead app greedily tries to serve all URLs >below code.python.org/. > >Yet another hint at the evil force-Bazaar-upon-them conspiracy . Naw, I think I just f'd up the Apache conf. I'll try to fix that. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkkKyLYACgkQ2YZpQepbvXG2pQCgh61b5hA7ECT5mNO+vbmJICYR tRoAoJEa3XGzM06qNepsnhHldzLPF4kG =8JK8 -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 31, 2008, at 08:58 AM, Barry Warsaw wrote: >Naw, I think I just f'd up the Apache conf. I'll try to fix that. And by "I'll" of course I meant "Martin". :) Thanks Martin! - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkkKy5MACgkQ2YZpQepbvXF8PgCdGou6DzXzBfOhZ4QvtJo3oQY4 QKMAmQECJQhxOxqLU1Og7K9OPVZ3VXvN =BdOC -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 31, 2008, at 03:50 AM, Eric Smith wrote: >I posted this yesterday about using bzr: > > >I'd like to try it out, but the instructions on > > http://www.python.org/dev/bazaar/ say to get wget > > http://code.python.org/snapshots/python-bzr-snapshot.tar.bz2, which is > > a 404. Anyone know the actual URL? Looks like this is fixed now too. Aside: Now that Python 2.6 isn't on the trunk, we need to add a bzr mirror for python26-maint. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkkK3RQACgkQ2YZpQepbvXFDYQCgolodW5cMVjGPUzLRyK2vsaHE 2dUAoKJzwXNVmyNBzgFSSgnMS4dPx7Mp =LD00 -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
> Looks like it should be > > http://code.python.org/loggerhead/static/images/ico_folder.gif I did a quick hack for the static files, but many of the links still don't work. loggerhead has specific support for reverse proxies (through loggerhead.conf), but I couldn't figure out how to activate that. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Has anyone made the argument for keeping the GIL to discourage threading? I'm only throwing this out there and I'm sure we'd want to improve things no matter what, but I would like to voice the concern anyway. We all know there are people who think threading is the answer to all things, and who don't understand the difficulties and problems with threading. Often, when trying to persuade them of alternative solutions, one selling point is "your python threads can't take advantage of multiple cores, anyway, because of the GIL." I worry, if it is ever actually removed, that I won't be able to convince these people that you really don't need to spawn twenty threads to write an IRC bot. On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Neil Schemenauer wrote: >> Sigurd Torkel Meldgaard <[EMAIL PROTECTED]> wrote: >>> For a student project in a course on virtual machines, we are >>> evaluating the possibility to experiment with removing the GIL >>> from CPython >> >> Hi, >> >> It's great to hear of this kind of project. I think what you want >> to do is difficult but possible. The major compilcation would be >> that extension modules would have to re-written since they all >> assume a reference counting GC. A foreign function interface like >> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it >> does place different demands on extension module authors. > > Michael Foord's comment about the way Ironclad does it suggests a > possible interim step that would allow existing extensions to be used at > the cost of some additional overhead: use free threading in the core, > but have an "extension module lock" that cuts things back to a single > thread whenever non-core code is invoked. > > It wouldn't be as fast as a completely free-threaded setup (since the > EML would need to operate as a variant of a 'read/write' lock, with the > core code mostly leaving it in the multiple-threads-allowed 'read' mode, > promoting it to single-thread-only 'write' mode when potentially > dispatching to non-core code, and dropping back to the 'read' mode when > the dispatch landed back in core code again). > > Over time, extension modules which were updated to use their own > internal locks could also drop back to free-threading mode. In essence, > the GIL would still exist, but instead of being a simple mutex the way > it is now, it would have an intermediate stage which allowed multiple > threads to continue running. Once most extension modules had been > converted, the function to promote the EML from multi-threaded mode to > single-thread only mode could be removed. > > A mechanism like that would avoid one of the more subtle problems in > removing the GIL: currently, CPython-specific Python code can assume > that invocation of a C function from Python code is an atomic operation, > since the body of the function executes as a single bytecode (we're > talking about the case here where the C function is known not to release > the GIL). > > I've written code which relied on the GIL in exactly that way, and I'm > sure I'm far from the only Python programmer to do so. If the GIL was > removed without adding an extension module lock, such applications would > require additional locking either in the C extension module or in the > Python code. > > A possible starting point for such a project: move the locking > mechanisms that are implemented in Python in the threading module to > C-based implementations in _thread so that a richer low overhead locking > arrangement is available to extensions modules (and the rest of the core). > > Cheers, > Nick. > > -- > Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia > --- > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
> A couple nits. Leaving off the trailing / yields a 404. (No big deal > though). More importantly, there seem to be no images, e.g.: > > http://code.python.org/static/images/ico_folder.gif > > Looks like it should be > > http://code.python.org/loggerhead/static/images/ico_folder.gif I'll leave these to Barry; the loggerhead installation is still not complete (so it's not yet an official service). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
Martin> I have now restored the original URL structure, and moved the Martin> loggerhead installation to Martin> http://code.python.org/loggerhead/ A couple nits. Leaving off the trailing / yields a 404. (No big deal though). More importantly, there seem to be no images, e.g.: http://code.python.org/static/images/ico_folder.gif Looks like it should be http://code.python.org/loggerhead/static/images/ico_folder.gif Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 07:59:01AM -0400, Calvin Spealman wrote: > Has anyone made the argument for keeping the GIL to discourage > threading? I haven't, but I would support such argument. In my humble opinion multithreading is rarely a good answer, and for multicore CPUs it's a wrong answer. For multicore CPUs the answer is multiprocessing + IPC (or a good database; filesystem is not a good database because it requires locking). One big problem with multithreading - it violates "divide et conqueror" principle. In programming almost everything is about separation of access rights - namespaces, modules, etc. Multithreading model allows a number of processes to share memory. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proper initialization of structs
Alexandre Vassalotti wrote: But what if PyType_GenericAlloc is used for tp_alloc? As far as I know, the memory block allocated with PyType_GenericAlloc is zeroed. Oh, you are right. The generic allocator uses memset to initialize the block with \0. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Nick Coghlan wrote: If I recall correctly, the main blocker to ctypes portability is libffi portability. So if anyone would like to see ctypes on more platforms, then that's the limitation they really need to attack. ctypes is also missing utilities to write code that works on 32 and 64bit platforms. Without a tool like http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious to avoid and fix segfaults. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Calvin Spealman wrote: Has anyone made the argument for keeping the GIL to discourage threading? I'm only throwing this out there and I'm sure we'd want to improve things no matter what, but I would like to voice the concern anyway. We all know there are people who think threading is the answer to all things, and who don't understand the difficulties and problems with threading. Often, when trying to persuade them of alternative solutions, one selling point is "your python threads can't take advantage of multiple cores, anyway, because of the GIL." There are many concurrency problems for which multiprocessing is *not* a good answer and for which careful use of threads is a relatively straightforward answer. For people with those sorts of problems CPython (and therefore the perception is Python) does not offer a usable concurrency story unless you are prepared to also write in C. Unfortunately many in the Python community seem pathologically opposed to even good use of threading and refuse to believe that such situations even exist. In the meantime, people with these problems (or who believe they are likely to have such problems) simply turn to other languages. All the best, Michael Foord I worry, if it is ever actually removed, that I won't be able to convince these people that you really don't need to spawn twenty threads to write an IRC bot. On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: Neil Schemenauer wrote: Sigurd Torkel Meldgaard <[EMAIL PROTECTED]> wrote: For a student project in a course on virtual machines, we are evaluating the possibility to experiment with removing the GIL from CPython Hi, It's great to hear of this kind of project. I think what you want to do is difficult but possible. The major compilcation would be that extension modules would have to re-written since they all assume a reference counting GC. A foreign function interface like CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it does place different demands on extension module authors. Michael Foord's comment about the way Ironclad does it suggests a possible interim step that would allow existing extensions to be used at the cost of some additional overhead: use free threading in the core, but have an "extension module lock" that cuts things back to a single thread whenever non-core code is invoked. It wouldn't be as fast as a completely free-threaded setup (since the EML would need to operate as a variant of a 'read/write' lock, with the core code mostly leaving it in the multiple-threads-allowed 'read' mode, promoting it to single-thread-only 'write' mode when potentially dispatching to non-core code, and dropping back to the 'read' mode when the dispatch landed back in core code again). Over time, extension modules which were updated to use their own internal locks could also drop back to free-threading mode. In essence, the GIL would still exist, but instead of being a simple mutex the way it is now, it would have an intermediate stage which allowed multiple threads to continue running. Once most extension modules had been converted, the function to promote the EML from multi-threaded mode to single-thread only mode could be removed. A mechanism like that would avoid one of the more subtle problems in removing the GIL: currently, CPython-specific Python code can assume that invocation of a C function from Python code is an atomic operation, since the body of the function executes as a single bytecode (we're talking about the case here where the C function is known not to release the GIL). I've written code which relied on the GIL in exactly that way, and I'm sure I'm far from the only Python programmer to do so. If the GIL was removed without adding an extension module lock, such applications would require additional locking either in the C extension module or in the Python code. A possible starting point for such a project: move the locking mechanisms that are implemented in Python in the threading module to C-based implementations in _thread so that a richer low overhead locking arrangement is available to extensions modules (and the rest of the core). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Le Friday 31 October 2008 14:13:01 Christian Heimes, vous avez écrit : > ctypes is also missing utilities to write code that works on 32 and > 64bit platforms. Without a tool like > http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious > to avoid and fix segfaults. I wrote some ctypes binding and some ctypes tools are missing. The most important lacks are: - get_errno() - size_t and types You may be interrested by my modules: get errno value: http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_errno.py (ctypes_support has a get_errno function) open the C library: http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_libc.py (ok, code is trivial, it's not critical to integrate it to ctypes) *size_t* and types! http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_stdint.py (my module is incomplete, some types are missing) Some functions to ease ctypes development: http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_tools.py - bytes2type() and bytes2array() are useful to convert a bytes array to a ctypes object Should I open issues? -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 9:17 AM, Michael Foord <[EMAIL PROTECTED]> wrote: > Calvin Spealman wrote: >> >> Has anyone made the argument for keeping the GIL to discourage >> threading? I'm only throwing this out there and I'm sure we'd want to >> improve things no matter what, but I would like to voice the concern >> anyway. We all know there are people who think threading is the answer >> to all things, and who don't understand the difficulties and problems >> with threading. Often, when trying to persuade them of alternative >> solutions, one selling point is "your python threads can't take >> advantage of multiple cores, anyway, because of the GIL." >> >> > > There are many concurrency problems for which multiprocessing is *not* a > good answer and for which careful use of threads is a relatively > straightforward answer. For people with those sorts of problems CPython (and > therefore the perception is Python) does not offer a usable concurrency > story unless you are prepared to also write in C. > > Unfortunately many in the Python community seem pathologically opposed to > even good use of threading and refuse to believe that such situations even > exist. In the meantime, people with these problems (or who believe they are > likely to have such problems) simply turn to other languages. > > All the best, > > Michael Foord There's room in python for both (all). Threading, as given to us by Java and others is hard to get right and is used by many because its "just there", but it is *still* useful for many people. If python were to have free threading, courtesy of a lack of the GIL, it would help those people quite a bit. Sometimes you just need shared state. Myself? I used multiprocess *and* threads all the time for various reasons. Of course: Does shared state make more sense as implemented via the Erlang green threads system - wherein threads are independent of one another, but pass messages between one another via the parent? (Van and I were talking about this for some time yesterday :)) But fundamentally - there's still an attraction to things like IPC/Multiprocessing, Actors, message passing/etc as well. I don't see free threading and other language improvements for additional styles of concurrent/distributed programming as mutually exclusive. Excluding the possibility of free-threading because "we don't like it" is sorta silly. It would be equivalent of blocking aynsc libraries for the same reason. All of that being said: I think there's room to aim for a *better* threading model, or even a series of higher level abstractions (on top of the basic abstractions threading provides on top of thread for example). For example: if you were building a Java application today, would you use the basic java.lang.thread classes and all of the other locking primitives, or would you go and use the abstractions and utilities in the java.util.concurrent package? Most people would reply with the latter: The language now provides to you a proven set of tools which have been "proven" to work, are "safe" - it doesn't make sense for them to go and build out their own thread factory and what not. Imagine a concurrent package for python: in it you have your basics - threading, multiprocessing (maybe "acting" too, but that name sucks, how about greenprocesses? :)) and you also have utilities like multiprocessing.pool, multiprocessing.manager/etc, threading.producer_consumer, etc. We build these out and make them "as safe as possible". Over time, we begin to hide more and more of the primitives and encourage usage of safer, proven abstractions. Take a look at something like Kamaelia - it's a dog pile of abstractors and metaphors that make it dead simple to do concurrent things without exposing the end-programmer to a lot of the underlying pain and basics. Sure: If I'm and end user I have to decide "threads, processes or green processes" but that's really an extension of the problem domain. Threads aren't good for some problems, processes are bad for others, etc. Just some pie-in-the-sky thoughts. Fundamentally, the GIL is not an end-user "feature" unless that end user is writing extension modules - and in that case, it's a pretty nice feature. However end users *benefit* from the GIL by having a rich standard library and a much more rich ecosystem of 3rd party modules - and that benefit can't be understated or sacrificed. -jesse ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Calvin> Has anyone made the argument for keeping the GIL to discourage Calvin> threading? I'm only throwing this out there and I'm sure we'd Calvin> want to improve things no matter what, but I would like to voice Calvin> the concern anyway. We all know there are people who think Calvin> threading is the answer to all things, and who don't understand Calvin> the difficulties and problems with threading. Often, when trying Calvin> to persuade them of alternative solutions, one selling point is Calvin> "your python threads can't take advantage of multiple cores, Calvin> anyway, because of the GIL." I think the "we are all consenting adults here" rubric applies. As Michael Foord indicated there are tasks for which multithreading is appropriate. If people want to shoot themselves in the foot with threads that's their prerogative. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Christian Heimes schrieb: > Nick Coghlan wrote: >> If I recall correctly, the main blocker to ctypes portability is libffi >> portability. So if anyone would like to see ctypes on more platforms, >> then that's the limitation they really need to attack. > > ctypes is also missing utilities to write code that works on 32 and > 64bit platforms. Without a tool like > http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious > to avoid and fix segfaults. There's a difference between 'ctypes does not compile or work on my platform' and 'how to write portable code with ctypes'. Of course I would very much welcome patches for the first issue, plus tools or other helpers for the second. -- Thanks, Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
Victor Stinner schrieb: > Le Friday 31 October 2008 14:13:01 Christian Heimes, vous avez écrit : >> ctypes is also missing utilities to write code that works on 32 and >> 64bit platforms. Without a tool like >> http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious >> to avoid and fix segfaults. > > I wrote some ctypes binding and some ctypes tools are missing. The most > important lacks are: > - get_errno() > - size_t and types > > You may be interrested by my modules: Some of the problems you list here have already been fixed in Python 2.6: > > get errno value: > http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_errno.py > (ctypes_support has a get_errno function) A new calling convention has been added for *safe* access to errno. > open the C library: > http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_libc.py > (ok, code is trivial, it's not critical to integrate it to ctypes) This code is not in ctypes, but with Python2.6 it will also work on windows. > *size_t* and types! > http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_stdint.py > (my module is incomplete, some types are missing) Well, the signed types are named c_int8, c_uint8, and so on. size_t is named c_size_t, please look into Lib/ctypes/__init__.py. IIRC, these types are present event longer. > Some functions to ease ctypes development: > http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_tools.py > - bytes2type() and bytes2array() are useful to convert a bytes array >to a ctypes object There are also new ctypes methods in Python 2.6 .from_buffer() and .from_buffer_copy() which should cover this use case. An example from your module: > 124 def bytes2type(bytes, type): > 125 """ > 126 Cast a bytes string to an objet of the specified type. > 127 """ > 128 return cast(bytes, POINTER(type))[0] > 129 Personally I would prefer to spell out 'cast(bytes, POINTER(type))[0]' instead of documenting and learning what bytes2type() does. > Should I open issues? > Please decide yourself. Thanks, Thomas ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 4:59 AM, Calvin Spealman <[EMAIL PROTECTED]> wrote: > Has anyone made the argument for keeping the GIL to discourage > threading? Oooh, you are on to my secret plan! :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My patches
> I have a similar list that I have been thinking about proposing. I did > a blog post about it at > http://sayspy.blogspot.com/2008/08/what-are-typical-steps-issue-goes.html > and received positive feedback: > > * triage > * verify bug > * test needed > * needs patch > * patch review > * commit review > * committed/rejected Would it be possible to add a new field (a combo box), let say "stage" (or "step")? But we need also some hooks to automate stage changes: * attach a new patch => set stage to "patch review" (and maybe add the keyword "review") * remove "patch reviewed" keyword => set stage to "commit review" * set status to "closed" => set stage to "commited/rejected" We can maybe use JavaScript for some operations (it's easy to catch a combobox change). About the old issues, is it possible to write a script setting the stage using the same rules (using the keywords and status)? -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My patches
> But we need also some hooks to automate stage changes: > * attach a new patch => set stage to "patch review" >(and maybe add the keyword "review") The latter already happens automatically. > * remove "patch reviewed" keyword => set stage to "commit review" > * set status to "closed" => set stage to "commited/rejected" > > We can maybe use JavaScript for some operations (it's easy to catch a > combobox > change). Automatic state changes are best done with roundup detectors - see the detectors directory of the python-dev instance to get an idea of what can be done (I used to think these things should be "react" actions, but making them "audit" actions actually looks nicer in the history) > About the old issues, is it possible to write a script setting the stage > using > the same rules (using the keywords and status)? Most certainly. See the scripts directory of the python-dev instance for examples. Regards, Martin P.S. These things might better belong to tracker-discuss. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (10/24/08 - 10/31/08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2153 open (+33) / 13915 closed (+19) / 16068 total (+52) Open issues with patches: 705 Average duration of open issues: 710 days. Median duration of open issues: 1951 days. Open Issues Breakdown open 2137 (+33) pending16 ( +0) Issues Created Or Reopened (52) ___ Miserable subprocess.Popen performance 10/24/08 http://bugs.python.org/issue4194created skip.montanaro Regression for executing packages10/24/08 http://bugs.python.org/issue4195created vajda patch library documentation errors 10/24/08 CLOSED http://bugs.python.org/issue4196created LambertDW Doctest module does not work with zipped packages10/24/08 http://bugs.python.org/issue4197created belopolsky patch os.path.normcase gets fooled on windows with mapped linux networ 10/24/08 http://bugs.python.org/issue4198created dawidjoubert add shorthand global and nonlocal statements 10/24/08 http://bugs.python.org/issue4199created benjamin.peterson patch atexit module not safe in Python 3.0 with multiple interpreters 10/24/08 CLOSED http://bugs.python.org/issue4200created grahamd patch, needs review Pdb cannot access source code in zipped packages.10/24/08 http://bugs.python.org/issue4201created belopolsky patch Multiple interpreters and readline module hook functions.10/25/08 http://bugs.python.org/issue4202created grahamd adapt sphinx-quickstart for windows 10/25/08 http://bugs.python.org/issue4203created timmie Cannot build _multiprocessing, math, mmap and readline of Python 10/25/08 http://bugs.python.org/issue4204created akitada patch unexpected str.__init__() behaviour on b'' 10/25/08 CLOSED http://bugs.python.org/issue4205created keldonin multiprocessing docs 10/26/08 http://bugs.python.org/issue4206created LambertDW Remove backwards compatibility in _heapq for performance 10/26/08 CLOSED http://bugs.python.org/issue4207created krisvale patch, patch, easy Make multiprocessing compatible with Python 2.4 and 2.5 10/26/08 http://bugs.python.org/issue4208created christian.heimes patch, patch from __future__ import unicode_literals, print_function doesn't 10/26/08 CLOSED http://bugs.python.org/issue4209created benjamin.peterson patch os.remove can be used to remove directories on SunOS 10/26/08 CLOSED http://bugs.python.org/issue4210created giampaolo.rodola frozen packages set an improper __path__ value 10/27/08 http://bugs.python.org/issue4211created brett.cannon email.LazyImporter does not use absolute imports 10/27/08 http://bugs.python.org/issue4212created brandonbloom Lower case file system encoding
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
>> Has anyone made the argument for keeping the GIL to discourage >> threading? > > Oooh, you are on to my secret plan! :-) I completely agree that there are other approaches to parallelism and concurrency that are much better than threading. However, I don't think this is a good argument for having poor support for parallel threads in Python (i.e. keeping the GIL). The reason is that threads are extremely useful (and often required) for implementing other approaches to concurrency, such as message passing. Two examples: Take Erlang for example. Erlang uses a shared nothing/message passing approach to concurrency (Yes!). However, Erlangs implementation of this approach relies heavily upon threads in the low-level of the interpreter. Without this usage of threads Erlang would be able to provide the multicore scalability that it does using message passing. Same is true of MPI. From the user's perspective MPI is just message passing. But, MPI implementations use threads internally extensively. Bottom line: threads may be a bad end in themselves (I agree with this), but they are a great means to better things. Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Fri, Oct 31, 2008 at 11:55 AM, Brian Granger <[EMAIL PROTECTED]> wrote: >>> Has anyone made the argument for keeping the GIL to discourage >>> threading? >> >> Oooh, you are on to my secret plan! :-) > > I completely agree that there are other approaches to parallelism and > concurrency that are much better than threading. However, I don't > think this is a good argument for having poor support for parallel > threads in Python (i.e. keeping the GIL). The reason is that threads > are extremely useful (and often required) for implementing other > approaches to concurrency, such as message passing. Two examples: > > Take Erlang for example. Erlang uses a shared nothing/message passing > approach to concurrency (Yes!). However, Erlangs implementation of > this approach relies heavily upon threads in the low-level of the > interpreter. Without this usage of threads Erlang would be able to > provide the multicore scalability that it does using message passing. > > Same is true of MPI. From the user's perspective MPI is just message > passing. But, MPI implementations use threads internally extensively. > > Bottom line: threads may be a bad end in themselves (I agree with > this), but they are a great means to better things. Despite the name, safethread has more to do with erlang than it does traditional threading. Alas, greenmicropseudothreadactormonitortaskfibreropebobprocess is a little too long for normal use, so I'm gonna keep calling it thread (and to distinguish it from an OS process). -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Has python-dev collapsed?
On Wed, Oct 29, 2008 at 9:22 PM, <[EMAIL PROTECTED]> wrote: > > On 29 Oct, 09:14 pm, [EMAIL PROTECTED] wrote: >> >> Will 3.1 and 2.7 also be parallel releases? (I ask, not having read >> the 3xxx PEPS at all.) >> >> If yes, why? While I can see a case for 2.6/3.0 being in sync -- new >> features in 2.6 ease the transition to 3.0 -- I'd imagine that 3.1 >> would be better with a shorter cycle (6-9 months) because there are >> more possible rough edges to clean up. 3.1 would likely include >> bugfixes from the eventual 2.7, so 3.1 might also trigger 2.6.1, but I >> don't think there's any harm if 3.1 contains features or incompatible >> fixes that are unavailable to 2.x users. > > 2.7 is where a new version of 2to3 will be distributed. As people actually > start trying to migrate real-world code, lots of issues are inevitably going > to be discovered there. From my perspective, 2.7 and 3.1 need to be synced > because 2.7 is the tool that will actually enable users of existing code to > *use* 3.1. FWIW, I'm also backporting new 2to3 features and bugfixes to the 2.6 series. -- Cheers, Benjamin Peterson "There's nothing quite as beautiful as an oboe... except a chicken stuck in a vacuum cleaner." ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
