Dear Experts, I am getting a MemoryError when creating a dict in a long running process and suspect this is due to memory fragmentation. Any suggestions would be welcome. Full details of the problem are below.
I have a long running processing which eventually dies to a MemoryError exception. When it dies, it is using roughly 900 MB on a 4 GB Windows XP machine running Python 2.5.4. If I do "import pdb; pdb.pm()" to debug, I see that it is dying inside a method when trying to create a dict with about 2000 elements. If I instead do something like p setattr(self,'q',list([list(xxx)+list(xxx)+list(xxx)+list(xxx) for xxx in self.data])) inside the debugger, I can make the memory increase to about 1.5 GB WITHOUT getting a memory error. If instead I do something like p setattr(self,'q',dict([(xxx,xxx) for xxx in range(1400)])) inside the debugger, I get a MemoryError exception. If instead I do something like p setattr(self,'q',dict([(xxx,xxx) for xxx in range(1300)])) inside the debugger, I get no Exception. I infer that python is trying to allocate a bunch of contiguous space for the dict and due to fragmentation it can't find the contiguous space and therefore it gives a memory error. 1. Does this sound plausible or could something else be causing the problem? 2. Does anyone have suggestions on how to fix this? Some time Googling brings up the following 2004 thread started by Evan Jones: http://mail.python.org/pipermail/python-dev/2004-October/049480.html but I'm unable to find a solution to the problem I'm having. Any help would be much appreciated. Thanks, -E -- http://mail.python.org/mailman/listinfo/python-list
