Re: problem with multiprocessing and defaultdict

2010-01-12 Thread Wolodja Wentland
On Tue, Jan 12, 2010 at 11:48 +0100, wiso wrote: > They sent back the object filled with data. The problem is very simple: I > have a container, the container has a method read(file_name) that read a > huge file and fill the container with datas. I have more then 1 file to read > so I want to pa

Re: problem with multiprocessing and defaultdict

2010-01-12 Thread wiso
Robert Kern wrote: > On 2010-01-11 17:50 PM, wiso wrote: > >> The problem now is this: >> start reading file r1_200909.log >> start reading file r1_200910.log >> readen 488832 lines from file r1_200910.log >> readen 517247 lines from file r1_200909.log >> >> with huge file (the real case) the pro

Re: problem with multiprocessing and defaultdict

2010-01-11 Thread Robert Kern
On 2010-01-11 17:50 PM, wiso wrote: The problem now is this: start reading file r1_200909.log start reading file r1_200910.log readen 488832 lines from file r1_200910.log readen 517247 lines from file r1_200909.log with huge file (the real case) the program freeze. Is there a solution to avoid

Re: problem with multiprocessing and defaultdict

2010-01-11 Thread wiso
Robert Kern wrote: > On 2010-01-11 17:15 PM, wiso wrote: >> I'm using a class to read some data from files: >> >> import multiprocessing >> from collections import defaultdict >> >> def SingleContainer(): >> return list() >> >> >> class Container(defaultdict): >> """ >> this class s

Re: problem with multiprocessing and defaultdict

2010-01-11 Thread Robert Kern
On 2010-01-11 17:15 PM, wiso wrote: I'm using a class to read some data from files: import multiprocessing from collections import defaultdict def SingleContainer(): return list() class Container(defaultdict): """ this class store odd line in self["odd"] and even line in self["

problem with multiprocessing and defaultdict

2010-01-11 Thread wiso
I'm using a class to read some data from files: import multiprocessing from collections import defaultdict def SingleContainer(): return list() class Container(defaultdict): """ this class store odd line in self["odd"] and even line in self["even"]. It is stupid, but it's only a