Hello, I'm having the same problem. I'm creating two dictionaries with about 700,000 to 1,600,000 keys each, each key pointing to a smaller structure of embedded dictionaries. On a 8 x 3.0GHz Xeon Mac Pro with 17GB of RAM, cPickle is excruciatingly slow. Moreover, it crashes with memory error (12). I've been looking for alternatives, even thinking of writing up my own file format and read/write functions. As a last resort I tried dumping the two lists with the marshal module and it worked great. A 200MB and a 300MB file are created in a couple of seconds without a crash.
You should be warned that marshal is not meant as a general data-persistence module. But as long as you have a rather simple data structure and you are using it with a specific python version, it seems to run circles around cPickle. Hope it helps. -- View this message in context: http://www.nabble.com/writing-large-dictionaries-to-file-using-cPickle-tp21708938p23246693.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
