Re: memory use with regard to large pickle files

2008-10-18 Thread Aaron Brady
Catherine Moroney wrote: > I'm writing a python program that reads in a very large > "pickled" file (consisting of one large dictionary and one > small one), and parses the results out to several binary and hdf > files. > > The program works fine, but the memory load is huge. The size of > the pi

Re: memory use with regard to large pickle files

2008-10-18 Thread Lawrence D'Oliveiro
In message <[EMAIL PROTECTED]>, Catherine Moroney wrote: > I'm writing a python program that reads in a very large > "pickled" file (consisting of one large dictionary and one > small one), and parses the results out to several binary and hdf > files. Job for a database? -- http://mail.python.org

Re: memory use with regard to large pickle files

2008-10-15 Thread Martin v. Löwis
> The program works fine, but the memory load is huge. The size of > the pickle file on disk is about 900 Meg so I would theoretically > expect my program to consume about twice that (the dictionary > contained in the pickle file plus its repackaging into other formats), > but instead my program n