Re: writing large dictionaries to file using cPickle

2009-01-30 Thread perfreem
On Jan 28, 6:08 pm, Aaron Brady wrote: > On Jan 28, 4:43 pm, perfr...@gmail.com wrote: > > > On Jan 28, 5:14 pm, John Machin wrote: > > > > On Jan 29, 3:13 am, perfr...@gmail.com wrote: > > > > > hello all, > > > > > i have a large dictionary which contains about 10 keys, each key has a > > > > v

Re: writing large dictionaries to file using cPickle

2009-01-28 Thread perfreem
On Jan 28, 5:14 pm, John Machin wrote: > On Jan 29, 3:13 am, perfr...@gmail.com wrote: > > > > > hello all, > > > i have a large dictionary which contains about 10 keys, each key has a > > value which is a list containing about 1 to 5 million (small) > > dictionaries. for example, > > > mydict = {

Re: writing large dictionaries to file using cPickle

2009-01-28 Thread perfreem
On Jan 28, 11:32 am, pyt...@bdurham.com wrote: > Hi, > > Change: > > pickle.dump(mydict, pfile) > > to: > > pickle.dump(mydict, pfile, -1 ) > > I think you will see a big difference in performance and also a much > smaller file on disk. > > BTW: What type of application are you developing that crea

writing large dictionaries to file using cPickle

2009-01-28 Thread perfreem
hello all, i have a large dictionary which contains about 10 keys, each key has a value which is a list containing about 1 to 5 million (small) dictionaries. for example, mydict = {key1: [{'a': 1, 'b': 2, 'c': 'hello'}, {'d', 3, 'e': 4, 'f': 'world'}, ...], key2: [...]} in total

Re: how to optimize object creation/reading from file?

2009-01-28 Thread perfreem
On Jan 28, 10:06 am, Bruno Desthuilliers wrote: > perfr...@gmail.com a écrit : > > > > > hi, > > > i am doing a series of very simple string operations on lines i am > > reading from a large file (~15 million lines). i store the result of > > these operations in a simple instance of a class, and t

how to optimize object creation/reading from file?

2009-01-28 Thread perfreem
hi, i am doing a series of very simple string operations on lines i am reading from a large file (~15 million lines). i store the result of these operations in a simple instance of a class, and then put it inside of a hash table. i found that this is unusually slow... for example: class myclass(o

scatterhist and resizing figures

2009-01-26 Thread perfreem
i am using scatterhist to plot some data. i find that when i plot the data and matlab shows it in the figure window, stretching the figure window (with the mouse) to enlarge it actually changes the properties of the figure. for example, making it bigger sometimes reveals more tick marks - like the

writing pickle function

2009-01-23 Thread perfreem
hello, i am using nested defaultdict from collections and i would like to write it as a pickle object to a file. when i try: from collections import defaultdict x = defaultdict(lambda: defaultdict(list)) and then try to write to a pickle file, it says: TypeError: can't pickle function objects