> Date: Sun, 01 Sep 2013 22:55:56 +0200
> From: "Pascal J. Bourguignon" <p...@informatimago.com>
> To: guile-user@gnu.org
>
> Darren Hoo <darren....@gmail.com> writes:
>
>> It is way too slow to read numbers from a file simply by using `read'
>>
>> for example a txt file contains 10,000,000 line of numbers:
>
> To get speed, I would:
>
> 1- open a binary file,
>
> 2- perform double-buffering I/O, (ie. read a buffer while you process
> another),
>
> 3- assume the input format is as specified, ie. no check.
> ?
> (let ((byte (vector-ref buffer i)))
> (if (= byte 10)
> (begin (set! values (cons value values))
> (set! value 0))
> (set! value (+ (* value 10) (mod byte 16)))))
> ?
>
>
> Actually, only using buffered I/O matters. For big files, what slows
> down is I/O, not the handling of characters or checking of the syntax or
> anything that's done in memory. If you want to read your data faster,
> keep it in RAM, or else in SSD.
I know this is not exactly a Scheme solution, but if you can choose your file
format, consider using HDF5. You can read directly into a typed (srfi4) array,
and a lot of languages have bindings (e.g. h5py), so it's a portable format.
I've found it to be fast.
I haven't found bindings for Guile on the web (the ones I use depend on local
stuff), but it shouldn't be a lot of work, and it's more flexible than a
one-off integer reader. Of course, if that's all you need, then disregard this.
If you or someone else is interested in doing proper bindings using the ffi, I
can lend a hand and test.
Daniel
