On Jan 28, 2:38 pm, ocsch...@gmail.com wrote: > Hi, all. > > I have an application that that creates, manipulates, and finally > archives on disk 10^6 instances of an object that in CS/DB terms is > best described as a relation. > > It has 8 members, all of them common Python datatypes. 6 of these are > set once and then not modified. 2 are modified around 4 times before > the instance's archving. Large collections (of small lists) of these > objects are created, iterated through, and sorted using any and all of > the 8 members as sorting keys. > > It neither has nor needs custom methods. > > I used a simple dictionary to create the application prototype. Now I > need to speed things up. > I first tried changing to a new style class, with __slots__, __init__, > __getstate__& __setstate__ (for pickling) and was shocked to see > things SLOW down over dictionaries. > > So of these options, where should I go first to satisfy my need for > speed? > > 0. Back to dict > 1. old style class > 2. new style class > 3. new style class, with __slots__, with or without some nuance I'm > missing. > 4. tuple, with constants to mark the indices > 5. namedTuple > 6. other...
Hello, quoting myself from another thread today: There is the 'shelve' module. You could create a shelf that tells you the filename of the 5 other ones. A million keys should be no problem, I guess. (It's standard library.) All your keys have to be strings, though, and all your values have to be pickleable. If that's a problem, yes you will need ZODB or Django (I understand), or another relational DB. There is currently no way to store live objects. -- http://mail.python.org/mailman/listinfo/python-list
