Re: [Python-Dev] [python] Re: New lines, carriage returns, and Windows
"Guido van Rossum" <[EMAIL PROTECTED]> wrote: > > Does anyone else have the feeling that discussions with Mr. MacLaren > don't usually bear any fruit? Yes. I do. My ability to predict the (technical) future is good; my ability to persuade people of it is almost non-existent. However, when an almost identical thread to this one occurs in a decade's time, I shall be safely retired. And, assuming no changes to the basic models, when one occurs in twenty years' time, I shall be forgotten. Such is life. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
On 02/10/2007, Adam Olsen <[EMAIL PROTECTED]> wrote: > > On 10/1/07, Greg Ewing <[EMAIL PROTECTED]> wrote: > > Justin Tulloss wrote: > > > Would > > > somebody care to give me a brief overview on how the current gc module > > > interacts with the interpreter > > > > The cyclic GC kicks in when memory is running low. Since > > This isn't true at all. It's triggered by heuristics based on the > total number of allocated objects. It doesn't know how much memory is > available and is not called if an allocation fails. Correct. And that reminds me of the limitation of the the Python GC: it doesn't take into account how much memory is being indirectly retained by a Python Object. Like in the example I already gave, gtk.gdk.Pixbuf can easily hold hundreds of megabytes, yet the GC gives it as much consideration as it does to a simple python integer object which is several orders of magnitude smaller. If someone wanted to improve the GC, that person should consider adding a protocol for the GC to retrieve the ammount of memory indirectly held by a python object, and take such memory into consideration in its heuristics. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
On Tue, 2007-10-02 at 10:50 +0100, Gustavo Carneiro wrote: > Correct. And that reminds me of the limitation of the the Python GC: > it doesn't take into account how much memory is being indirectly > retained by a Python Object. Like in the example I already gave, > gtk.gdk.Pixbuf can easily hold hundreds of megabytes, yet the GC gives > it as much consideration as it does to a simple python integer object > which is several orders of magnitude smaller. That sounds like a case for the Pixbuf object to have a "close" method (not necessarily called that) that releases the resources. The point of GC is that you normally don't care if memory is released sooner or later; for stuff you do care about, such as files, shared memory, or even large memory objects, there is always explicit management. cStringIO's "close" method provides a precedent. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
On 02/10/2007, Hrvoje Nikšić <[EMAIL PROTECTED]> wrote: > > On Tue, 2007-10-02 at 10:50 +0100, Gustavo Carneiro wrote: > > Correct. And that reminds me of the limitation of the the Python GC: > > it doesn't take into account how much memory is being indirectly > > retained by a Python Object. Like in the example I already gave, > > gtk.gdk.Pixbuf can easily hold hundreds of megabytes, yet the GC gives > > it as much consideration as it does to a simple python integer object > > which is several orders of magnitude smaller. > > That sounds like a case for the Pixbuf object to have a "close" method > (not necessarily called that) that releases the resources. The point of > GC is that you normally don't care if memory is released sooner or > later; for stuff you do care about, such as files, shared memory, or > even large memory objects, there is always explicit management. > cStringIO's "close" method provides a precedent. I think close in real files is needed not so much because you want to free memory, but that you want to prevent data loss by flushing the file buffer into the actual file at a certain point in time. And I suspect that cStringIO just added a close() method for compatibility with real files. But of course I speculating here... -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
On Tue, 2007-10-02 at 11:30 +0100, Gustavo Carneiro wrote: > even large memory objects, there is always explicit > management. > cStringIO's "close" method provides a precedent. > > I think close in real files is needed not so much because you want to > free memory, but that you want to prevent data loss by flushing the > file buffer into the actual file at a certain point in time. You can also do that with flush. What is specific for close is that it frees up the system resources occupied by the open file. Calling it at a known point in time ensures that these external resources aren't occupied any longer than necessary, regardless of the object deallocator's policies. That is why open(filename).read() is discouraged, despite being a useful idiom in throwaway scripts and working just fine in CPython. The fact that programmers need Pixbuf's memory released sooner rather than later might indicate that Pixbuf would benefit from a "close"-like method. In any case, I trust your judgment on Pixbufs, but merely point out that in general it is unwise to rely on the deallocator to release resources where timing of the release matters. > And I suspect that cStringIO just added a close() method for > compatibility with real files. But of course I speculating here... That could be the case, but then the method could also be a no-op. I only brought up cStringIO as a precedent for a close method that works with a memory-only object. (StringIO's close also tries to arrange for the buffer to be freed immediately, inasmuch as that is possible to achieve in pure Python.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
On 10/2/07, Hrvoje Nikšić <[EMAIL PROTECTED]> wrote: > > On Tue, 2007-10-02 at 10:50 +0100, Gustavo Carneiro wrote: > > Correct. And that reminds me of the limitation of the the Python GC: > > it doesn't take into account how much memory is being indirectly > > retained by a Python Object. Like in the example I already gave, > > gtk.gdk.Pixbuf can easily hold hundreds of megabytes, yet the GC gives > > it as much consideration as it does to a simple python integer object > > which is several orders of magnitude smaller. > > That sounds like a case for the Pixbuf object to have a "close" method > (not necessarily called that) that releases the resources. The point of > GC is that you normally don't care if memory is released sooner or > later; for stuff you do care about, such as files, shared memory, or > even large memory objects, there is always explicit management. > cStringIO's "close" method provides a precedent. Agreed, objects that consume a lot of resources of any type should have a method to clean themselves up or the language should take special notice of 'del myObject' and call the destructor immediately if nothing else refers to the object at that point rather than leaving it up to a GC. Its just good programming practice to do such things on "large" objects regardless of the underlying allocation implementation. -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python tickets summary
2007/9/19, Ron Adam <[EMAIL PROTECTED]>: > I noticed that there is a background of light blue between marks. That is > hard to see on my computer because it is so close to the grey tone. Made it a little darker, now it's easier to look. > Also shouldn't the light blue background bar extend all the way to the end > for all open items? No, because this "light blue bar" is the span of time from opened to last comment. Note that these items are *all* open. Thanks for the feedback! Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Backporting Decimal
People: I don't decide myself what to do in this case. The Decimal module appeared in 2.4, and received just slight modifications for 2.5. Since it appeared, a "just download and use it" version was available for Python 2.3 users. But for 2.6, it was fully renewed. Not only was updated to the last spec from Cawlishaw, but also a lot of (very slightly, arithmetic corner cases) bugs were fixed. Also, it received a lot of code simplifications and speedups. All this is, in most part, because Mark Dickinson was involved here, and he did (and is doing) a great work. When 2.6 is out, I'll update that downloadable for Py2.3. But what to do with Py2.5? Decimal is a pretty stand alone module, and I'm absolutely sure that just backporting the whole module and its testcases will fix a lot of problems, and Py2.5 users will have new functionality, but is this ok? (of course, I can not be 100% sure that any bugs were introduced in these modifications). Recommendations? Thank you very much! -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
> Decimal is a pretty stand alone module, and I'm absolutely sure that > just backporting the whole module and its testcases will fix a lot of > problems, and Py2.5 users will have new functionality, but is this ok? Yes! We have guaranteed that spec updates are to be treated as bug fixes and backported. This is especially important in this case because other errors have been fixed and the test cases have grown. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
2007/10/2, Raymond Hettinger <[EMAIL PROTECTED]>: > Yes! We have guaranteed that spec updates are to be treated as bug fixes and > backported. This is especially important in this case > because other errors have been fixed and the test cases have grown. Perfect! I'll backport it to 2.5... what about 2.4? Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
>> Yes! We have guaranteed that spec updates are to be treated as bug fixes >> and backported. This is especially important in this >> case >> because other errors have been fixed and the test cases have grown. > > Perfect! I'll backport it to 2.5... what about 2.4? If there are any plans for another 2.4 release, then yes; otherwise, why bother. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
Jeremy covered already most of it, so I'll only address specific points: > and I think that the current gc > module is of the mark-and-sweep variety. That is incorrect. It's two phases, but in the first phase, it isn't "mark", but "count", and in the second phase, it's not "sweep" but "break". To "count" means to count the number of references to an object, and to "break" means to break the cycles at selected places, expecting that reference counting will actually cause objects to become released. > Is the trend going to be to > move away from reference counting and towards the mark-and-sweep > implementation that currently exists, or is reference counting a firmly > ingrained tradition? For the CPython implementation, there is no trend. The language spec explicitly states that automatic memory management is unspecified and implementation-defined, yet people rely on reference counting in many programs. That is not the reason to kick it out, though - it stays around because a true mark-and-sweep algorithm cannot be implemented on top of the current object API (as Jeremy explains, there is no notion of root objects in the VM). > On a more immediately relevant note, I'm not certain I understand the > full extent of the gc module. From what I've read, it sounds like it's > fairly close to a fully functional GC, yet it seems to exist only as a > cycle-detecting backup to the reference counting mechanism. Would > somebody care to give me a brief overview on how the current gc module > interacts with the interpreter, or point me to a place where that is > done? What is "this" that is done? The collector? I think you found Modules/gcmodule.c already. Take particular notice of the tp_traverse and tp_clear type methods. > Why isn't the mark-and-sweep mechanism used for all memory > management? See above - it's not implementable, because the root objects get not tracked. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
> If there are any plans for another 2.4 release, then yes; otherwise, why > bother. Please don't make any functional changes to 2.4. There may be additional releases, but (IMO) they should be security releases only, i.e. contain no functional changes whatsoever (unless they fix a security issue). I'm still trying to work on the PEP that formalizes this notion. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
On 10/2/07, Facundo Batista <[EMAIL PROTECTED]> wrote: > > 2007/10/2, Raymond Hettinger <[EMAIL PROTECTED]>: > > > Yes! We have guaranteed that spec updates are to be treated as bug > fixes and backported. This is especially important in this case > > because other errors have been fixed and the test cases have grown. > > Perfect! I'll backport it to 2.5... what about 2.4? There's one potential breakage that I see here: the Decimal.__pow__ function has different semantics in the new version of Decimal (nonintegers are now allowed as exponents; some previous restrictions on argument sizes no longer exist), so backporting the new version of __pow__ might cause difficulties. In particular, some cases of three-argument pow that previously worked (giving arguably nonsensical results) will now raise an exception. To be honest, I'd be quite surprised to find that *anyone* was using the three-argument pow, but perhaps we should be careful here? Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
Martin v. Löwis <[EMAIL PROTECTED]> wrote: >> Why isn't the mark-and-sweep mechanism used for all memory >> management? > > See above - it's not implementable, because the root objects get not > tracked. To further elaborate, the main obstacle is with extension modules. Most of them create roots and there is no defined API for the Python interpreter to find them. Perhaps a more serious problem is that CPython and 3rd party extension modules rely heavily on the fact that the GC does not move objects. There are GC strategies that perform well when faced with high object allocation rates but, AFAIK, all of them rely on moving objects. Neil ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting Decimal
2007/10/2, Mark Dickinson <[EMAIL PROTECTED]>: > difficulties. In particular, some cases of three-argument pow that > previously worked (giving arguably nonsensical results) will now raise an > exception. To be honest, I'd be quite surprised to find that *anyone* was If previously it gave a arguably nonsensical results, and now it raises an exception, I consider it a fixed bug, :) -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
Hrvoje Nikšić wrote: > That sounds like a case for the Pixbuf object to have a "close" method > (not necessarily called that) that releases the resources. The point of > GC is that you normally don't care if memory is released sooner or > later; I think the problem here is that the GC's lack of knowledge about how much memory is being used means that you need to care more than you should have to. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GC Changes
> To further elaborate, the main obstacle is with extension modules. > Most of them create roots and there is no defined API for the Python > interpreter to find them. That is a problem, but furthermore, I feel that local variables stored in stack frames of threads are even more difficult to integrate. For the extension-module globals, some sort of registration could be done (and PEP 3121 provides an infrastructure for that). For stack frames, such a registration is difficult to make efficient. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
