"Marcus Goldfish" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
>I need to write a "fast" file reader in python for binary files structured
>as:
>
> … x[0] y[0] z[0] x[1] y[1] z[1] …
>
> where c[k] is the k-th element from sequence c. As mentioned, the
> file is binary -- spaces above are just for visualization. Each
> element, c[k], is a 16-bit int.
You have a sequence of machine-format triples
> (ii) how can I handle the 16-bit word aspect of the binary data?
that you want to unpack (see struct module) and unzip (your code).
> I can assume I know the number of sequences in the file a priori.
Then you can pre-allocate x,y,z instead of growing them by appends:
x = n*[0]; ...
> Files are stored and processed on a
> WinXP machine (in case Endian-ness matters).
I believe struct module takes care of this. Read its doc.
> (i) should I use generators for iterating over the file?
If you will only ever unpack when you also unzip to separate lists, then
you might put the unpack code in the inner loop of the unzip method. If
you think that you might someday directly process the file as a sequence of
scaler triples without creating intermediate x,y,z lists, then you should
factor out the file read/unpack as you are thinking.
> (iii) ultimately, the data will need to be processed in chunks of
> M-values at a time... I assume this means I need some
> form of buffered io wrapper, but I'm not sure where to start
> with this.
I presume this means M triplets or 3*N values.
In any case, Python is ahead of you ;-).
>>>help(file.read)
read([size]) -> read at most size bytes, returned as a string.
If the size argument is negative or omitted, read until EOF is reached.
Terry J. Reedy
--
http://mail.python.org/mailman/listinfo/python-list
