When I read in files containing a lot of data I don't format the files
to be python code but just data, then write a python function to parse
the input. Mostly my data files are not so large as yours, but I have
processed larger ones such as the ones at
https://math.dartmouth.edu/~jvoight/hmf.html which are in fact files
designed to be read in by Magma (just as yours for GAP), but we have
Sage/python code to read it in line by line with no problem. For
example the file data_6_01997632 there is this big:
$ wc data_6_01997632
91 426892 141419775 data_6_01997632
(i.e. 140MB) and reading it in is instantaneous (before doing any processing):
sage: f = open("data_6_01997632")
sage: %time for L in f.readlines(): n=len(L)
CPU times: user 28 ms, sys: 48 ms, total: 76 ms
Wall time: 72.9 ms
Your job would then be to write your data in a way which makes parsing
the input relatively easy.
John
On 16 September 2017 at 10:13, Simon King <simon.k...@uni-jena.de> wrote:
> Hi!
>
> I have a large file (2.7*10^6 lines, 204.5*10^6 bytes) of code
> that defines a python dict, some dict values are matrices
> of dimension roughly 800x1200 over GF(8), some dict values are
> other dicts.
>
> Problem: When I try to load the file with sage.repl.load.load,
> my laptop very soon starts swapping.
>
> The data file is a Python readable translation of a libgap
> readable data file. libgap only needs 30 seconds to read the
> corresponding data. Thus, it *is* possible to read the data in
> reasonable time.
>
> Is there a faster way to read and evaluate a large python code
> block than sage.repl.load.load?
>
> Best regards,
> Simon
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-devel@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
