While Kurt gave some excellent ideas for using numpy, there were
some missing details in your original post that might help folks
come up with a "work smarter, not harder" solution.
Clearly, you're not loading it into memory just for giggles --
surely you're *doing* something with it once it's in memory.
With details of what you're trying to do with that data, some of
the smart-minds on the list may be able to provide a
solution/algorithm that doesn't require having everything in
memory concurrently.
Or you may try streaming through your data-sources, pumping it
into a sqlite/mysql/postgres database, allowing for more
efficient querying of the data. Both mysql & postgres offer the
ability to import data directly into the server without the need
for (or overhead of) a bajillion INSERT statements which may also
speed up the slurping-in process.
-tkc
--
http://mail.python.org/mailman/listinfo/python-list
