On Sun, 12 Feb 2006 05:11:02 -0800, MKoool wrote: > I have an application with one function called "compute", which given a > filename, goes through that file and performs various statistical > analyses. It uses arrays extensively and loops alot. it prints the > results of it's statistical significance tests to standard out. Since > the compute function returns and I think no variables of global scope > are being used, I would think that when it does, all memory returns > back to the operating system.
I may be mistaken, and if so I will welcome the correction, but Python does not return memory to the operating system until it terminates. Objects return memory to Python when they are garbage collected, but not the OS. > Instead, what I see is that every iteration uses several megs more. > For example, python uses 52 megs when starting out, it goes through > several iterations and I'm suddenly using more than 500 megs of ram. > > Does anyone have any pointers on how to figure out what I'm doing > wrong? How big is the file you are reading in? If it is (say) 400 MB, then it is hardly surprising that you will be using 500MB of RAM. If the file is 25K, that's another story. How are you storing your data while you are processing it? I'd be looking for hidden duplicates. I suggest you re-factor your program. Instead of one giant function, break it up into lots of smaller ones, and call them from compute. Yes, this will use a little more memory, which might sound counter-productive at the moment when you are trying to use less memory, but in the long term it will allow your computer to use memory more efficiently (it is easier to page small functions as they are needed than one giant function), and it will be much easier for you to write and debug when you can isolate individual pieces of the task in individual functions. Re-factoring will have another advantage: you might just find the problem on your own. -- Steven. -- http://mail.python.org/mailman/listinfo/python-list
