On Thu, Jun 3, 2010 at 3:43 PM, Emin.shopper Martinian.shopper < emin.shop...@gmail.com> wrote:
> Dear Experts, > > I am getting a MemoryError when creating a dict in a long running > process and suspect this is due to memory fragmentation. Any > suggestions would be welcome. Full details of the problem are below. > > I have a long running processing which eventually dies to a > MemoryError exception Here's something good about Python memory leaks: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks <http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks>Just curious: What if you use a gdbm key-value store or treap instead of a builtin {} dictionary? gdbm is usually built in, and my treap module is here, as pure python or partial CPython+partial cython: http://stromberg.dnsalias.org/~dstromberg/treap/ The gdbm module would require your keys and values to be strings (unless you wrap it), but it should pretty much take dictionary-related Virtual Memory (VM) out of the picture by mostly storing your data to disk. One way of wrapping one would be to use my (UCI's) cachedb module, which does implicit to- and from-string conversions using functions you provide to its constructor, and allows you to control how many values to cache in VM: http://stromberg.dnsalias.org/~strombrg/cachedb.html The treap module should be able to store arbitrary objects in VM out of the box, and is much like a dictionary but implemented quite differently - its chief difference is a little memory _overhead_ (normally, maybe not this time due to the significantly different implementation details) in order to get the "dictionary's" objects always sorted in key order, and to store things as a (mostly!) self-balanced tree (it sacrifices fastidious balancing to get better average performance) with a heap invariant. Because it stores things as a balanced tree of discrete objects, it should be pretty clear that it's not looking for contiguous memory unless the backend CPython memory management does for some reason beyond dictionary specifics.
-- http://mail.python.org/mailman/listinfo/python-list
