On 5/21/05, DJTB <[EMAIL PROTECTED]> wrote: > [posted to comp.lang.python, mailed to [EMAIL PROTECTED]
[Following up to both places.] > I'm having problems storing large amounts of objects in a ZODB. > After committing changes to the database, elements are not cleared from > memory. Since the number of objects I'd like to store in the ZODB is too > large to fit in RAM, my program gets killed with signal 11 or signal 9... The problem here is a common one with a first attempt at using ZODB. The problem is that ZODB manages memory at the granularity of first-class persistent objects -- that is, instances of classes that inherit from Persistent. ZODB can move such objects in and out of memory at transaction boundaries, which allows your application to use many more objects than it has physical memory for. It looks like your application has a single persistent instance -- the root ExtendedTupleTable -- so there's no way for ZODB to manage the memory. That object and everything reachable from it must be in memory at all times. You need to re-structure the program so that is has more first-class persistent objects. If, for example, the ExtendedTuple objects inherited from Persistent, then they could reside on disk except when you are manipulating them. The ObjectInterning instance is another source of problem, because it's a dictionary that has an entry for every object you touch. The various other dictionaries in your program will also be memory hogs in they have very many entries. The typical way to structure a ZODB program is to use one of the BTrees implementation types instead of a dictionary, because the BTree does not keep all its keys and values in memory at one time. (Its internal organization is a large collection of first-class persistent objects representing the BTree buckets and internal tree nodes.) You must use some care with BTrees, because the data structure maintains a total ordering on the keys. (And a dictionary does not.) The ZODB/ZEO programming guide has a good section on BTrees here: http://www.zope.org/Wikis/ZODB/guide/node6.html Jeremy -- http://mail.python.org/mailman/listinfo/python-list
