On 2013-06-15, Peter Otten wrote:
> Giorgos Tzampanakis wrote:
>
>> I have a program that saves lots (about 800k) objects into a shelve
>> database (I'm using sqlite3dbm for this since all the default python dbm
>> packages seem to be unreliable and effectively unusable, but this is
>> another discussion).
>>
>> The process takes about 10-15 minutes. During that time I see memory usage
>> steadily rising, sometimes resulting in a MemoryError. Now, there is a
>> chance that my code is keeping unneeded references to the stored objects,
>> but I have debugged it thoroughly and haven't found any.
>>
>> So I'm beginning to suspect that the pickle module might be keeping an
>> internal cache of objects being pickled. Is this true?
>
> Pickler/Unpickler objects use a cache to maintain object identity, but at
> least shelve in the standard library uses a new Pickler/Unpickler for each
> set/get operation.
>
> I don't have sqlite3dbm, but you can try the following:
>
>>>> import shelve
>>>> class A: pass
> ...
>>>> a = A()
>>>> s = shelve.open("tmp.shelve")
>>>> s["x"] = s["y"] = a
>>>> s["x"] is s["y"]
> False
This returns False in my case.
> If you are getting True there must be a cache. One way to enable a cache
> yourself is writeback:
No, I haven't enabled writeback.
--
Real (i.e. statistical) tennis and snooker player rankings and ratings:
http://www.statsfair.com/
--
http://mail.python.org/mailman/listinfo/python-list
