On 23-Oct-2012 18:17, Paul Rubin wrote:
Virgil Stokes <v...@it.uu.se> writes:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a performance hit to scanning pointers
backwards in memory, due to cache misprediction. If it's something
you're just running a few times, seeking backwards the simplest
approach. If you're really trying to optimize the thing, you might
buffer up large chunks (like 1 MB) before writing. If you're writing
once and reading multiple times, you might reverse the order of records
within the chunks during the writing phase.
I am writing (forward) once and reading (backward) once.
You're of course taking a performance bath from writing the program in
Python to begin with (unless using scipy/numpy or the like), enough that
it might dominate any effects of how the files are written.
I am currently using SciPy/NumPy
Of course (it should go without saying) that you want to dump in a
binary format rather than converting to decimal.
Yes, I am doing this (but thanks for "underlining" it!)
Thanks Paul :-)
--
http://mail.python.org/mailman/listinfo/python-list
