I have a program that saves lots (about 800k) objects into a shelve database (I'm using sqlite3dbm for this since all the default python dbm packages seem to be unreliable and effectively unusable, but this is another discussion).
The process takes about 10-15 minutes. During that time I see memory usage steadily rising, sometimes resulting in a MemoryError. Now, there is a chance that my code is keeping unneeded references to the stored objects, but I have debugged it thoroughly and haven't found any. So I'm beginning to suspect that the pickle module might be keeping an internal cache of objects being pickled. Is this true? -- Real (i.e. statistical) tennis and snooker player rankings and ratings: http://www.statsfair.com/ -- http://mail.python.org/mailman/listinfo/python-list
