Re: processing a Very Large file

2005-05-19 Thread Gregory Bond
Mike Meyer wrote: > > Unknown. Python relies on the C alloc/free routines for handling > memory. del may immediately free() the memory (I don't know), but that > doesn't mean it gets released to the OS. None of the implementations > of alloc/free I'm aware of ever give the memory back to the OS.

Re: processing a Very Large file

2005-05-18 Thread Mike Meyer
DJTB <[EMAIL PROTECTED]> writes: > I'm not a Python memory specialist, but does del immediately release/free > the memory to the OS? I thought it was impossible to let Python immediately > release memory. Unknown. Python relies on the C alloc/free routines for handling memory. del may immediately

Re: processing a Very Large file

2005-05-18 Thread DJTB
Tim Peters wrote: > >>tuple_size = int(splitres[0])+1 >>path_tuple = tuple(splitres[1:tuple_size]) >>conflicts = Set(map(int,splitres[tuple_size:-1])) > > Do you really mean to throw away the last value on the line? That is, > why is the slice here [tuple_size:-1] rather

RE: processing a Very Large file

2005-05-18 Thread DJTB
Robert Brewer wrote: > DJTB wrote: >> I'm trying to manually parse a dataset stored in a file. The >> data should be converted into Python objects. >> > > The first question I would ask is: what are you doing with "result", and > can the consumption of "result" be done iteratively? > > The pr

Re: processing a Very Large file

2005-05-17 Thread Gregory Bond
DJTB wrote: > Hi, > > I'm trying to manually parse a dataset stored in a file. The data should be > converted into Python objects. In addition to what the others have mentioned, this sort of problem is pretty easy to do with a C coded extension type, if you have (or can buy/borrow) any C skills

Re: processing a Very Large file

2005-05-17 Thread Steve M
I'm surprised you didn't recommend to use ZODB. Seems like an ideal way to manage this large amount of data as a collection of Python objects... -- http://mail.python.org/mailman/listinfo/python-list

RE: processing a Very Large file

2005-05-17 Thread Robert Brewer
DJTB wrote: > I'm trying to manually parse a dataset stored in a file. The > data should be converted into Python objects. > > Here is an example of a single line of a (small) dataset: > > 3 13 17 19 -626177023 -1688330994 -834622062 -409108332 > 297174549 955187488 > 589884464 -1547848504 8573

Re: processing a Very Large file

2005-05-17 Thread Tim Peters
[DJTB] > I'm trying to manually parse a dataset stored in a file. The data should be > converted into Python objects. > > Here is an example of a single line of a (small) dataset: > > 3 13 17 19 -626177023 -1688330994 -834622062 -409108332 297174549 955187488 > 589884464 -1547848504 857311165 585