Possible to set cpython heap size?
I have an application that scans and processes a bunch of text files. The content I'm pulling out and holding in memory is at least 200MB. I'd love to be able to tell the CPython virtual machine that I need a heap of, say 300MB up front rather than have it grow as needed. I've had a scan through the archives of comp.lang.python and the python docs but cannot find a way to do this. Is this possible to configure the PVM this way? Much appreciated, Andy -- -- http://mail.python.org/mailman/listinfo/python-list
Re: Possible to set cpython heap size?
> Why do you want that? And no, it is not possible. And to be honest: I have > no idea why e.g. the JVM allows for this. > > Diez The reason why is simply that I know roughly how much memory I'm going to need, and cpython seems to be taking a fair amount of time extending its heap as I read in content incrementally. Ta, Andy -- -- http://mail.python.org/mailman/listinfo/python-list
Re: Possible to set cpython heap size?
On Feb 22, 10:53 am, a bunch of folks wrote: > Memory is basically free. This is true if you are simply scanning a file into memory. However, I'm storing the contents in some in-memory data structures and doing some data manipulation. This is my speculation: Several small objects per scanned line get allocated, and then unreferenced. If the heap is relatively small, GC has to do some work in order to make space for subsequent scan results. At some point, it realises it cannot keep up and has to extend the heap. At this point, VM and physical memory is committed, since it needs to be used. And this keeps going on. At some point, GC will take a good deal of time to compact the heap, since I and loading in so much data and creating a lot of smaller objects. If I could have a heap that is larger and does not need to be dynamically extended, then the Python GC could work more efficiently. Interesting discussion. Cheers, Andy -- -- http://mail.python.org/mailman/listinfo/python-list
