[Python-Dev] Re: Re: Re: Re: Patch review: [ 1094542 ] add Bunch type to collections module
Steven Bethard wrote: > That sounds reasonable to me. I'll fix update to be a staticmethod. > If people want other methods, I'll make sure they're staticmethods > too.[1] > > Steve > > [1] In all the cases I can think of, staticmethod is sufficient -- the > methods don't need to access any attributes of the Bunch class. If > anyone has a good reason to make them classmethods instead, let me > know... Great. I probably meant staticmethod. I don't use either much, so I don't really know the difference in the terminology. For a long time I stuck to 2.1 features for ipython and my other codes, and I seem to recall those appeared in 2.2. But you got what I meant :) Cheers, f ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python Interpreter Thread Safety?
Due to the issue of thread safety in the Python memory allocator, I have been wondering about thread safety in the rest of the Python interpreter. I understand that the interpreter is not thread safe, but I'm not sure that I have seen a discussion of the all areas where this is an issue. Here are the areas I know of: 1. The memory allocator. 2. Reference counts. 3. The cyclic garbage collector. 4. Current interpreter state is pointed to by a single shared pointer. 5. Many modules may not be thread safe (?). Ignoring the issue of #5 for the moment, are there any other areas where this is a problem? I'm curious about how much work it would be to allow concurrent execution of Python code. Evan Jones Note: One of the reasons I am asking is that my memory allocator patch is that it changes the current allocator from "sort of" thread safe to obviously unsafe. One way to eliminate this issue is to make the allocator completely thread safe, but that would require some fairly significant changes to avoid a major performance penalty. However, if it was one of the components that permitted the interpreter to go multi-threaded, then it would be worth it. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: [PyCon] Reg: Registration
Aahz, writing as [EMAIL PROTECTED], wrote: It's still January 28 here -- register now! I don't know if we'll be able to extend the registration price beyond that. Just in case anybody else might be wondering when the early bird registration deadline is, I've asked the registration team to allow the early bird price as long as it's January 28th somewhere in the world. There have been rumors that Guido will not be attending PyCon this year. I am happy to scotch them by pointing out that Guido van Rossum's keynote address will be on its traditional Thursday morning. I look forward to joining you all to hear Guido speak on "The State of Python". regards Steve -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/ Holden Web LLC +1 703 861 4237 +1 800 494 3119 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting rid of unbound methods: patch available
Guido van Rossum wrote: Here's a patch that gets rid of unbound methods, as discussed here before. A function's __get__ method now returns the function unchanged when called without an instance, instead of returning an unbound method object. I thought the main reason for existence of unbound methods (for user-defined classes, at least) was so that if you screw up a super call by forgetting to pass self, or passing the wrong type of object, you get a more helpful error message. I remember a discussion about this some years ago, in which you seemed to think the ability to produce this message was important enough to justify the existence of unbound methods, even though it meant you couldn't easily have static methods (this was long before staticmethod() was created). Have you changed your mind about that? Also, surely unbound methods will still have to exist for C methods? Otherwise there will be nothing to ensure that C code is getting the object type it expects for self. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting rid of unbound methods: patch available
[Guido] > > Here's a patch that gets rid of unbound methods, as > > discussed here before. A function's __get__ method > > now returns the function unchanged when called without > > an instance, instead of returning an unbound method object. [Greg] > I thought the main reason for existence of unbound > methods (for user-defined classes, at least) was so that > if you screw up a super call by forgetting to pass self, > or passing the wrong type of object, you get a more > helpful error message. Yes, Tim reminded me of this too. But he said he could live without it. :-) > I remember a discussion about this some years ago, in > which you seemed to think the ability to produce this > message was important enough to justify the existence > of unbound methods, even though it meant you couldn't > easily have static methods (this was long before > staticmethod() was created). > > Have you changed your mind about that? After all those years, I think the added complexity of unbound methods doesn't warrant having the convenience of the error message. > Also, surely unbound methods will still have to exist > for C methods? Otherwise there will be nothing to ensure > that C code is getting the object type it expects for > self. No, C methods have their own object type for that (which is logically equivalent to an unbound method). But there was a use case for unbound methods having to do with C methods of classic classes, in the implementation of built-in exceptions. Anyway, it's all moot because I withdrew the patch, due to the large amount of code that would break due to the missing im_class attribute -- all fixable, but enough not to want to break it all when 2.5 comes out. So I'm salting the idea up for 3.0. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Josiah Carlson wrote: While it seems that super() is the 'modern paradigm' for this, I have been using base.method(self, ...) for years now, and have been quite happy with it. I too would be very disappointed if base.method(self, ...) became somehow deprecated. Cooperative super calls are a different beast altogether and have different use cases. In fact I'm having difficulty finding *any* use cases at all for super() in my code. I thought I had found one once, but on further reflection I changed my mind. And I have found that the type checking of self provided by unbound methods has caught a few bugs that would probably have produced more mysterious symptoms otherwise. But I can't say for sure whether they would have been greatly more mysterious -- perhaps not. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Tim Peters wrote: I expect that's because he stopped working on Zope code, so actually thinks it's odd again to see a gazillion methods like: class Registerer(my_base): def register(*args, **kws): my_base.register(*args, **kws) I second that! My PyGUI code is *full* of __init__ methods like that, because of my convention for supplying initial values of properties as keyword arguments. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch review: [ 1009811 ] Add missing types to__builtin__
Jeff Rush wrote: If they are put into __builtins__, the documentation won't need updating. ;-) In that case, I'd rather prefer to correct the documentation. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
Evan Jones wrote: Due to the issue of thread safety in the Python memory allocator, I have been wondering about thread safety in the rest of the Python interpreter. I understand that the interpreter is not thread safe, but I'm not sure that I have seen a discussion of the all areas where this is an issue. Here are the areas I know of: This very much depends on the definition of "thread safe". If this mean "working correctly in the presence of threads", then your statement is wrong - the interpreter *is* thread-safe. The global interpreter lock (GIL) guarantees that only one thread at any time can execute critical operations. Since all threads acquire the GIL before performing such operations, the interpreter is thread-safe. 1. The memory allocator. 2. Reference counts. 3. The cyclic garbage collector. 4. Current interpreter state is pointed to by a single shared pointer. This is all protected by the GIL. 5. Many modules may not be thread safe (?). Modules often release the GIL through BEGIN_ALLOW_THREADS, if they know that would be safe if another thread would enter the Python interpreter. Ignoring the issue of #5 for the moment, are there any other areas where this is a problem? I'm curious about how much work it would be to allow concurrent execution of Python code. Define "concurrent". Webster's offers 1. operating or occurring at the same time Clearly, on a single-processor system, no two activities can execute concurrently - the processor can do at most one activity at any point in time. Perhaps you are asking whether it would be possible to change the current coarse-grained lock into a more finer-grained lock (as working without locks is not implementable). This is also known as "free threading". There have been attempts to implement free threading, and they have failed. Note: One of the reasons I am asking is that my memory allocator patch is that it changes the current allocator from "sort of" thread safe to obviously unsafe. The allocator is thread-safe in the presence of the GIL - you are supposed to hold the GIL before entering the allocator. Due to some unfortunate historical reasons, there is code which enters free() without holding the GIL - and that is what the allocator specifically deals with. Except for this single case, all callers of the allocator are required to hold the GIL. However, if it was one of the components that permitted the interpreter to go multi-threaded, then it would be worth it. Again, the interpreter supports multi-threading today. Removing the GIL is more difficult, though - nearly any container object (list, dictionary, etc) would have to change, plus the reference counting (which would have to grow atomic increment/decrement). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
On Jan 28, 2005, at 18:24, Martin v. Löwis wrote: 5. Many modules may not be thread safe (?). Modules often release the GIL through BEGIN_ALLOW_THREADS, if they know that would be safe if another thread would enter the Python interpreter. Right, I guess that the modules already have to deal with being reentrant and thread-safe, since Python threads could already cause issues. Ignoring the issue of #5 for the moment, are there any other areas where this is a problem? I'm curious about how much work it would be to allow concurrent execution of Python code. Define "concurrent". Webster's offers Sorry, I really meant *parallel* execution of Python code: Multiple threads simultaneously executing a Python program, potentially on different CPUs. There have been attempts to implement free threading, and they have failed. What I was trying to ask with my last email was what are the trouble areas? There are probably many that I am unaware of, due to my unfamiliarity the Python internals. Due to some unfortunate historical reasons, there is code which enters free() without holding the GIL - and that is what the allocator specifically deals with. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Again, the interpreter supports multi-threading today. Removing the GIL is more difficult, though - nearly any container object (list, dictionary, etc) would have to change, plus the reference counting (which would have to grow atomic increment/decrement). Wouldn't it be up to the programmer to ensure that accesses to shared objects, like containers, are serialized? For example, with Java's collections, there are both synchronized and unsynchronized versions. Evan Jones ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
Evan Jones wrote: Sorry, I really meant *parallel* execution of Python code: Multiple threads simultaneously executing a Python program, potentially on different CPUs. There cannot be parallel threads on a single CPU - for threads to be truly parallel, you *must* have two CPUs, at a minimum. Python threads can run truly parallel, as long as one of them invoke BEGIN_ALLOW_THREADS. What I was trying to ask with my last email was what are the trouble areas? There are probably many that I am unaware of, due to my unfamiliarity the Python internals. I think nobody really remembers - ask Google for "Python free threading". Greg Stein did the patch, and the main problem apparently was that the performance became unacceptable - apparently primarily because of dictionary locking. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Why do you think so? (I see in your previous post that you claim it is not completely correct, but I don't see any proof). Wouldn't it be up to the programmer to ensure that accesses to shared objects, like containers, are serialized? In a truly parallel Python, two arbitrary threads could access the same container, and it would still work. If some containers cannot be used simultaneously in multiple threads, this might ask for a desaster. For example, with Java's collections, there are both synchronized and unsynchronized versions. I don't think this approach can apply to Python. Python users are used to completely thread-safe containers, and lots of programs would break if the container would suddenly throw exceptions. Furthermore, the question is what kind of failure you'ld expect if an unsynchronized dictionary is used from multiple threads. Apparently, Java guarantees something (e.g. that the interpreter won't crash) but even this guarantee would be difficult to make. For example, for lists, the C API allows direct access to the pointers in the list. If the elements of the list could change in-between, an object in the list might go away after you got the pointer, but before you had a chance to INCREF it. This would cause a crash shortly afterwards. Even if that was changed to always return a new refence, lots of code would break, as it would create large memory leaks (code would have needed to decref the list items, but currently doesn't - nor is it currently necessary). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
On Sat, 2005-01-29 at 00:24 +0100, "Martin v. Löwis" wrote: > Evan Jones wrote: [...] > The allocator is thread-safe in the presence of the GIL - you are > supposed to hold the GIL before entering the allocator. Due to some > unfortunate historical reasons, there is code which enters free() > without holding the GIL - and that is what the allocator specifically > deals with. Except for this single case, all callers of the allocator > are required to hold the GIL. Just curious; is that "one case" a bug that needs fixing, or is the some reason this case can't be changed to use the GIL? Surely making it mandatory for all free() calls to hold the GIL is easier than making the allocator deal with the one case where this isn't done. I like the GIL :-) so much so I'd like to see it visible at the Python level. Then you could write your own atomic methods in Python. BTW, if what Evan is hoping for concurrent threads running on different processors in a multiprocessor system, then don't :-) It's been a while since I looked at multiprocessor architectures, but I believe threading's shared memory paradigm will always be hard to distribute efficiently over multiple CPU's. If you want to run on multiple processors, use processes, not threads. -- Donovan Baarda <[EMAIL PROTECTED]> http://minkirri.apana.org.au/~abo/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Python Interpreter Thread Safety?
Martin v. Löwis writes: > Due to some > unfortunate historical reasons, there is code which enters free() > without holding the GIL - and that is what the allocator specifically > deals with. Except for this single case, all callers of the allocator > are required to hold the GIL. Donovan Baarda writes: > Just curious; is that "one case" a bug that needs fixing, or is the some > reason this case can't be changed to use the GIL? Surely making it > mandatory for all free() calls to hold the GIL is easier than making the > allocator deal with the one case where this isn't done. What Martin is trying to say here is that it _IS_ mandatory to hold the GIL when calling free(). However, there is some very old code in existance (written by other people) which calls free() without holding the GIL. We work very hard to provide backward compatibility, so we are jumping through hoops to ensure that even this old code which is violating the rules doesn't get broken. -- Michael Chermside ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
... [Evan Jones] > What I was trying to ask with my last email was what are the trouble > areas? There are probably many that I am unaware of, due to my > unfamiliarity the Python internals. Google on "Python free threading". That's not meant to be curt, it's just meant to recognize that the task is daunting and has been discussed often before. [Martin v. Löwis] >> Due to some unfortunate historical reasons, there is code which enters >> free() without holding the GIL - and that is what the allocator specifically >> deals with. > Right, but as said in a previous post, I'm not convinced that the > current implementation is completely correct anyway. Sorry, I haven't had time for this. From your earlier post: > For example, is it possible to call PyMem_Free from two threads > simultaneously? Possible but not legal; undefined behavior if you try. See the "Thread State and the Global Interpreter Lock" section of the Python C API manual. ... only the thread that has acquired the global interpreter lock may operate on Python objects or call Python/C API functions There are only a handful of exceptions to the last part of that rule, concerned with interpreter and thread startup and shutdown, and they're explicitly listed in that section. The memory-management functions aren't among them. In addition, it's not legal to call PyMem_Free regardless unless the pointer passed to it was originally obtained from another function in the PyMem_* family (that specifically excludes memory obtained from a PyObject_* function). In a release build, all of the PyMem_* allocators resolve directly to the platform malloc or realloc, and all PyMem_Free has to determine is that they *were* so allocated and thus call the platform free() directly (which is presumably safe to call without holding the GIL). The hacks in PyObject_Free (== PyMem_Free) are there solely so that question can be answered correctly in the absence of holding the GIL. "That question" == "does pymalloc control the pointer passed to me, or does the system malloc?". In return, that hack is there solely because in much earlier versions of Python extension writers got into the horrible habit of allocating object memory with PyObject_New but releasing it with PyMem_Free, and because indeed Python didn't *have* a PyObject_Free function then. Other extension writers were just nuts, mixing PyMem_* calls with direct calls to system free/malloc/realloc, and ignoring GIL issues for all of those. When pymalloc was new, we went to insane lengths to avoid breaking that stuff, but enough is enough. > Since the problem is that threads could call PyMem_Free without > holding the GIL, it seems to be that it is possible. Yes, but not specific to PyMem_Free. It's clearly _possible_ to call _any_ function from multiple threads without holding the GIL. > Shouldn't it also be supported? No. If what they want is the system malloc/realloc/free, that's what they should call. > In the current memory allocator, I believe that situation can lead to > inconsistent state. Certainly, but only if pymalloc controls the memory blocks. If they were actually obtained from the system malloc, the only part of pymalloc that has to work correctly is the Py_ADDRESS_IN_RANGE() macro. When that returns false, the only other thing PyObject_Free() does is call the system free() immediately, then return. None of pymalloc's data structures are involved, apart from the hacks ensuring that the arena of base addresses is safe to access despite potentlly current mutation-by-appending. > ... > Basically, if a concurrent memory allocator is the requirement, It isn't. The attempt to _exploit_ the GIL by doing no internal locking of its own is 100% deliberate in pymalloc -- it's a significant speed win (albeit on some platforms more than others). > then I think some other approach is necessary. If it became necessary, that's what this section of obmalloc is for: SIMPLELOCK_DECL(_malloc_lock) #define LOCK() SIMPLELOCK_LOCK(_malloc_lock) #define UNLOCK()SIMPLELOCK_UNLOCK(_malloc_lock) #define LOCK_INIT() SIMPLELOCK_INIT(_malloc_lock) #define LOCK_FINI() SIMPLELOCK_FINI(_malloc_lock) You'll see that PyObject_Free() calls LOCK() and UNLOCK() at appropriate places already, but they have empty expansions now. Back to the present: [Martin] >> Again, the interpreter supports multi-threading today. Removing >> the GIL is more difficult, though - nearly any container object >> (list, dictionary, etc) would have to change, plus the reference >> counting (which would have to grow atomic increment/decrement). [Evan] > Wouldn't it be up to the programmer to ensure that accesses to shared > objects, like containers, are serialized? For example, with Java's > collections, there are both synchronized and unsynchronized versions. Enormous mounds of existing threaded Python code freely manipulates lists and dicts without explicit locking now. We can't break t
Re: [Python-Dev] Python Interpreter Thread Safety?
On Jan 28, 2005, at 20:27, Tim Peters wrote: The hacks in PyObject_Free (== PyMem_Free) are there solely so that question can be answered correctly in the absence of holding the GIL. "That question" == "does pymalloc control the pointer passed to me, or does the system malloc?". Ah! *Now* I get it. And yes, it will be possible to still support this in my patched version of the allocator. It just means that I have to leak the "arenas" array just like it did before, and then do some hard thinking about memory models and consistency to decide if the "arenas" pointer needs to be volatile. When pymalloc was new, we went to insane lengths to avoid breaking that stuff, but enough is enough. So you don't think we need to bother supporting that any more? Back to the present: Wouldn't it be up to the programmer to ensure that accesses to shared objects, like containers, are serialized? For example, with Java's collections, there are both synchronized and unsynchronized versions. Enormous mounds of existing threaded Python code freely manipulates lists and dicts without explicit locking now. We can't break that -- and wouldn't want to. Writing threaded code is especially easy (a relative stmt, not absolute) in Python because of it. Right, because currently Python switches threads on a granularity of opcodes, which gives you this serialization with the cost of never having parallel execution. Evan Jones ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
On Jan 28, 2005, at 19:44, Martin v. Löwis wrote: Python threads can run truly parallel, as long as one of them invoke BEGIN_ALLOW_THREADS. Except that they are really executing C code, not Python code. I think nobody really remembers - ask Google for "Python free threading". Greg Stein did the patch, and the main problem apparently was that the performance became unacceptable - apparently primarily because of dictionary locking. Thanks, I found the threads discussing it. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Why do you think so? (I see in your previous post that you claim it is not completely correct, but I don't see any proof). There are a number of issues actually, but as Tim points, only if the blocks are managed by PyMalloc. I had written a description of three of them here, but they are not relevant. If the issue is calling PyMem_Free with a pointer that was allocated with malloc() while PyMalloc is doing other stuff, then no problem: That is possible to support, but I'll have to think rather hard about some of the issues. For example, for lists, the C API allows direct access to the pointers in the list. If the elements of the list could change in-between, an object in the list might go away after you got the pointer, but before you had a chance to INCREF it. This would cause a crash shortly afterwards. Even if that was changed to always return a new refence, lots of code would break, as it would create large memory leaks (code would have needed to decref the list items, but currently doesn't - nor is it currently necessary). Ah! Right. In Java, the collections are all actually written in Java, and run on the VM. Thus, when some concurrent weirdness happens, it just corrupts the application, not the VM. However, in Python, this could actually corrupt the interpreter itself, crashing the entire thing with a very ungraceful Segmentation Fault or something similar. Evan Jones ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
