Mark Dickinson <dicki...@gmail.com> added the comment: Thanks for the extra information; that helps a lot.
I think this is expected behaviour: Python really does need that much memory to parse the file (as a Python file). Partly this is because Python objects actually do take up a fair amount of space: a length-4 list of floats on my (64-bit) machine takes 200 bytes, though on 32-bit machine this number should be a bit smaller. But partly it's that the compilation stage itself uses a lot of memory: for example, each of the floats in your input gets put into a dict during compilation; this dict is used to recognize multiple references to the same float, so that only one float object needs to be created for each distinct float value. And those dicts are going to get pretty big. I don't think that storing huge amounts of data in a .py file like this is usual practice, so I'm not particularly worried that importing a huge .py file can cause a MemoryError. For your case, I'd suggest parsing your datafile manually: reading the file line by line from within Python. Suggest closing this issue as "won't fix". ---------- nosy: +mark.dickinson _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue9180> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
