Re: [Python-Dev] Problems with the Python Memory Manager
Travis Oliphant wrote: > Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) > --- that's what the "array scalars" are: new types created in C. are you allocating PyTypeObject structures dynamically? why are you creating an awful lot of new type objects to represent the contents of a homogenous array? > If they don't get properly collected then that would definitely have > created the problem. It would seem this should be advertised when > telling people to use PyObject_New for allocating new memory for > an object. PyObject_New creates a new instance of a given type; it doesn't, in itself, create a new type. at this point, your description doesn't make much sense. more information is definitely needed... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problems with the Python Memory Manager
Travis Oliphant wrote: > The fact that it did happen is what I'm reporting on. If nothing will > be done about it (which I can understand), at least this thread might > help somebody else in a similar situation track down why their Python > process consumes all of their memory even though their objects are being > deleted appropriately. since that doesn't happen in other applications, I'm not sure this thread will help much -- unless you can provide us with enough details to figure out what makes this case so much different... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problems with the Python Memory Manager
Travis Oliphant <[EMAIL PROTECTED]> writes: > Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) > --- that's what the "array scalars" are: new types created in C. Ah! And, er, why? > If they don't get properly collected then that would definitely have > created the problem. types do get collected -- but only after the cycle collector has run. If you can still reproduce the problem can you try again but calling 'gc.set_threshold(1)'? > It would seem this should be advertised when telling people to use > PyObject_New for allocating new memory for an object. Nevertheless, I think it would be good if pymalloc freed its arenas. I think the reasons it doesn't are because of worries that people might be called PyObject_Free without holding the GIL, but that's been verboten for several years now so we can probably just let them suffer. I think there's even a patch on SF to do this... Cheers, mwh -- The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offence. -- Edsger W. Dijkstra, SIGPLAN Notices, Volume 17, Number 5 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problems with the Python Memory Manager
>> >> Bingo. Yes, definitely allocating new _types_ (an awful lot of >> them...) >> --- that's what the "array scalars" are: new types created in C. > > > Do you really mean that someArray[1] will create a new type to represent > the second element of someArray? I would guess that you create an > instance of a type defined in your extension. O.K. my bad. I can see that I was confusing in my recent description and possibly misunderstood the questions I was asked. It can get confusing given the dynamic nature of Python. The array scalars are new statically defined (in C) types (just like regular Python integers and regular Python floats). The ndarray is also a statically defined type. The ndarray holds raw memory interpreted in a certain fashion (very similar to Python's array module). Each ndarray can have a certain data type. For every data type that an array can be, there is a corresponding "array scalar" type. All of these are statically defined types. We are only talking about instances of these defined types. When the result of a user operation with an ndarray is a scalar, an instance of the appropriate "array scalar" type is created and passed back to the user. Previously we were using PyObject_New in the tp_alloc slot and PyObject_Del in the tp_free slot of the typeobject structure in order to create and destroy the memory for these instances. In this particular application, the user ended up creating many, many instances of these array scalars and then deleting them soon after. Despite the fact that he was not retaining any references to these scalars (PyObject_Del had been called on them), his application crawled to a halt after only several hunderd iterations consuming all of the available system memory. To verify that indeed no references were being kept, I did a detailed analysis of the result of sys.getobjects() using a debug build of Python. When I replaced PyObject_New (with malloc and PyObject_Init) and PyObject_Del (with free) for the "array scalars" types in scipy core, the users memory problems magically disappeared. I therefore assume that the problem is the memory manager in Python. Initially, I thought this was the old problem of Python not freeing memory once it grabs it. But, that should not have been a problem here, because the code quickly frees most of the objects it creates and so Python should have been able to re-use the memory. So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are always the same size. In practice, the memory was apparently not being re-used but instead new blocks were being allocated to handle the load. His code is quite complicated and it is difficult to replicate the problem. I realize this is not helpful for fixing the Python memory manager, and I wish I could be more helpful. However, replacing PyObject_New with malloc does solve the problem for us and that may help anybody else in this situation in the future. Best regards, -Travis ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Iterating a closed StringIO
Currently StringIO.StringIO and cStringIO.StringIO behave differently
when iterating a closed stream:
s = StringIO.StringIO("foo")
s.close()
s.next()
gives StopIteration, but
s = cStringIO.StringIO("foo")
s.close()
s.next()
gives "ValueError: I/O operation on closed file".
Should they raise the same exception? Should this be fixed for 2.5?
Bye,
Walter Dörwald
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On 11/16/05, Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Thomas Lee wrote: > > > Even if it meant we had just one function call - one, safe function call > > that deallocated all the memory allocated within a function - that we > > had to put before each and every return, that's better than what we > > have. > > alloca? > > (duck) > But how widespread is its support (e.g., does Windows have it)? -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On 11/16/05, Thomas Lee <[EMAIL PROTECTED]> wrote:
> Just messing around with some ideas. I was trying to avoid the ugly
> macros (note my earlier whinge about a learning curve) but they're the
> cleanest way I could think of to get around the problem without
> resorting to a mass deallocation right at the end of the AST run. Which
> may not be all that bad given we're going to keep everything in-memory
> anyway until an error occurs ... anyway, anyway, I'm getting sidetracked :)
>
> The idea is to ensure that all allocations within a single function are
> made using the pool so that a function finishes what it starts. This
> way, if the function fails it alone is responsible for cleaning up its
> own pool and that's all. No funkyness needed for sequences, because each
> member of the sequence belongs to the pool too. Note that the stmt_ty
> instances are also allocated using the pool.
>
> This breaks interfaces all over the place though. Not exactly a pretty
> change :) But yeah, maybe somebody smarter than I will come up with
> something a bit cleaner.
>
> --
>
> /* snip! */
>
> #define AST_SUCCESS(pool, result) return result
> #define AST_FAILURE(pool, result) asdl_pool_free(pool); return result
>
This is actually exactly what I was thinking of; macros that handle
returns and specify whether the return signals a success or failure.
One tweak I would do is posibly lock down the the variable name with
AST_POOL_ALLOC() at the start of a function that creates _arena_pool.
That way you don't need to pass in the specific pool. I don't see why
we will need to have multiple pools within a function. This also
allows the VISIT_* macros to be easily modified and not suddenly
require another argument to specify the arena name.
And all of this is easy to police since you can grep for 'return' and
make sure that it is meant to be there and not in actuality be one of
the macros. Basically gives us the mini-language that Nick mentioned
way back at the beginning of this thread.
Oh, and tweak the macros to be within ``do { ... } while(0)`` (``if
(1) AST_FAILURE(pool, NULL);`` will not expand properly otherwise).
-Brett
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Iterating a closed StringIO
On 11/17/05, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> Currently StringIO.StringIO and cStringIO.StringIO behave differently
> when iterating a closed stream:
>
> s = StringIO.StringIO("foo")
> s.close()
> s.next()
>
> gives StopIteration, but
>
> s = cStringIO.StringIO("foo")
> s.close()
> s.next()
>
> gives "ValueError: I/O operation on closed file".
>
> Should they raise the same exception? Should this be fixed for 2.5?
I think cStringIO is doing the right thing; "real" files behave the same way.
Submit a patch for StringIO (also docs please) and assign it to me and
I'll make sure it goes in.
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On Nov 17, 2005, at 12:46 PM, Brett Cannon wrote: ... >> alloca? >> >> (duck) >> > > But how widespread is its support (e.g., does Windows have it)? Yep, spelled with a leading underscore: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ vclib/html/_crt__alloca.asp Alex ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coroutines (PEP 342)
On 11/14/05, Bruce Eckel <[EMAIL PROTECTED]> wrote: > I just finished reading PEP 342, and it appears to follow Hoare's > Communicating Sequential Processes (CSP) where a process is a > coroutine, and the communicaion is via yield and send(). It seems that > if you follow that form (and you don't seem forced to, pythonically), > then synchronization is not an issue. > > What is not clear to me, and is not discussed in the PEP, is whether > coroutines can be distributed among multiple processors. If that is or > isn't possible I think it should be explained in the PEP, and I'd be > interested in know about it here (and ideally why it would or wouldn't > work). It seems to me that the concept of coroutines and PEP342 has very little to do with concurrency itself, apart from the fact that the generators form very convenient units of parallelization if you're willing to do some scheduling of them yourself, and only *potentially* with concurrency, i.e. only if you wrote a scheduler that supports running generator iterations concurrently on two processors. Otherwise there is no concurrency abstraction, unlike threads: it's cooperative and you clearly can see in the code the points where "switching" occurs (next(... ), yield ...). Please beat me with a stick if this is lunatic... ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Iterating a closed StringIO
Am 17.11.2005 um 22:03 schrieb Guido van Rossum:
> On 11/17/05, Walter Dörwald <[EMAIL PROTECTED]> wrote:
>> Currently StringIO.StringIO and cStringIO.StringIO behave differently
>> when iterating a closed stream:
>>
>> s = StringIO.StringIO("foo")
>> s.close()
>> s.next()
>>
>> gives StopIteration, but
>>
>> s = cStringIO.StringIO("foo")
>> s.close()
>> s.next()
>>
>> gives "ValueError: I/O operation on closed file".
>>
>> Should they raise the same exception? Should this be fixed for 2.5?
>
> I think cStringIO is doing the right thing; "real" files behave the
> same way.
>
> Submit a patch for StringIO (also docs please) and assign it to me and
> I'll make sure it goes in.
http://www.python.org/sf/1359365
Doc/lib/libstringio.tex only states "See the description of file
objects for operations", so I'm not sure how to update the
documentation.
Bye,
Walter Dörwald
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
Portability may also be an issue to take into consideration: http://www.eskimo.com/~scs/C-faq/q7.32.html http://archives.neohapsis.com/archives/postfix/2001-05/1305.html Cheers, Tom Alex Martelli wrote: >On Nov 17, 2005, at 12:46 PM, Brett Cannon wrote: >... > > >>>alloca? >>> >>>(duck) >>> >>> >>> >>But how widespread is its support (e.g., does Windows have it)? >> >> > >Yep, spelled with a leading underscore: >http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ >vclib/html/_crt__alloca.asp > > >Alex > >___ >Python-Dev mailing list >[email protected] >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/krumms%40gmail.com > > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Iterating a closed StringIO
On 11/17/05, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> Am 17.11.2005 um 22:03 schrieb Guido van Rossum:
>
> > On 11/17/05, Walter Dörwald <[EMAIL PROTECTED]> wrote:
> >> Currently StringIO.StringIO and cStringIO.StringIO behave differently
> >> when iterating a closed stream:
> >>
> >> s = StringIO.StringIO("foo")
> >> s.close()
> >> s.next()
> >>
> >> gives StopIteration, but
> >>
> >> s = cStringIO.StringIO("foo")
> >> s.close()
> >> s.next()
> >>
> >> gives "ValueError: I/O operation on closed file".
> >>
> >> Should they raise the same exception? Should this be fixed for 2.5?
> >
> > I think cStringIO is doing the right thing; "real" files behave the
> > same way.
> >
> > Submit a patch for StringIO (also docs please) and assign it to me and
> > I'll make sure it goes in.
>
> http://www.python.org/sf/1359365
Thanks!
> Doc/lib/libstringio.tex only states "See the description of file
> objects for operations", so I'm not sure how to update the
> documentation.
OK, so that's a no-op.
I hope there isn't anyone here who believes this patch would be a bad idea?
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory management in the AST parser & compiler
On Nov 17, 2005, at 5:00 PM, Thomas Lee wrote: > Portability may also be an issue to take into consideration: Of course -- but so is anno domini... the eskimo.com FAQ is (C) 1995, and the neohapsis.com page just points to the eskimo.com one: > http://www.eskimo.com/~scs/C-faq/q7.32.html > http://archives.neohapsis.com/archives/postfix/2001-05/1305.html In 2006, I'm not sure the need to avoid alloca is anywhere as strong. Sure, it could be wrapped into a LOCALLOC macro (with a companion LOCFREE one), the macro expanding to alloca/nothing on systems which do have alloca and to malloc/free elsewhere -- this would keep the sources just as cluttered, but still speed things up where feasible. E.g., on my iBook, a silly benchmark just freeing and allocating 80,000 hunks of 1000 bytes takes 13ms with alloca, 57 without... Alex ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
