Hi Bernard,
I'd certainly like to add support for binary files, but as I haven't had a
need for it myself I haven't had a good place to start.
As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste
together multiple mmap's to cover files that are larger than 2GB.
So if a record ended up spanning two mmap()'s, you couldn't return the raw
data as a single object without copying it into a new buffer first.
Also, if you provide a fixed record size in bytes for "doing the idx offset
maths", why do you need the end idx for the current line as well?
For example if you say file.bin is full of records each 100B in size, and
you ask for the 10th record; don't you already know that the length of the
record is 100B?
Hope I can help,
Alan
On Fri, Mar 8, 2013 at 1:44 AM, bernardH <un.compte.pour.tes...@gmail.com>wrote:
> Hi,
>
>
> On Wednesday, March 6, 2013 2:53:26 PM UTC+1, Alan Busby wrote:
>>
>>
>> With the release of Clojure 1.5 and it's new reducers, I figured this
>> would be a good time to release a library to help with file IO for
>> reducers. As reducers can only operate in parallel over specific
>> collections, like vec, it requires some work to use them against files.
>>
>> Iota wraps a text files using Java NIO's mmap() so you can treat files
>> larger than memory like a vector. Clojure's normal collection functions
>> (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in
>> parallel for fold.
>>
>
> Thanks, I already love your lib !
> However, in order to squeeze the last bits of performance, what I'd really
> need would to be directly access the raw data as byte arrays.
> The ideal API would provide my function with things looking and quaking
> like bytes arrays but doing the idx offset maths to directly access the
> maped data (read-only ), the next-to ideal would provide my function the
> ref to whole bytes array and give me the start and end idx for current line.
>
> (I'd like to port a Java code where the most significant performance boost
> was avoiding Strings creation when at all possible, which was possible by
> only peeking at most of the lines).
>
> Cheers,
>
> B.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
