The library defines two protocols, ByteSource and ByteSink. They expose a
'take-bytes!' and 'send-bytes!' method, respectively. An unoptimized
transfer involves shuttling bytes between these two methods, which in most
cases means one copy and one allocation (ByteBuffer can act as a ByteSource
and requires neither).
An optimized transform is anything which short-circuits this default
mechanism. Unfortunately, exactly how optimal this is depends heavily on
what you're transferring. Copying a File to a Channel representing a file
descriptor will often use OS-provided zero copy mechanisms, which makes it
a lot more "optimal" than the optimized transfer between an InputStream and
an OutputStream, which just involves reusing an array for shuttling data
between them. Basically, 'optimized-transfer?' returning true means that a
special function exists that is as optimal as I (or someone else who's
written a def-transfer) know how to make it.
The number of steps is definitely not the number of copies. Wrapping an
array in a ByteArrayInputStream, for instance, is just allocating a
wrapper. Some are more complicated though; creating an array from a
ByteBuffer is only zero-copy if the ByteBuffer is non-direct *and* the
position and limit haven't been moved. Right now my traversal is treating
each transform as equally complex, but an obvious improvement is to allow
each transform to be annotated with some sort of overhead metric, and
minimize based on that. A lot of the low-level optimizations are
implementation-specific, though, so it's always going to be a little fuzzy.
If you have ideas on any of the above, please speak up.
Zach
On Sunday, June 30, 2013 8:46:47 PM UTC-7, Dan Burkert wrote:
>
> Very cool, I've got a couple of question: the readme references optimized
> transfers, what qualifies as an optimized transfer? Also, would it be
> possible for byte-streams to give an estimation of the number of memory
> copies that happen in a given conversion (maybe this is as simple as the
> number of steps...)? Thanks for releasing this!
>
> -- Dan
>
> On Saturday, June 29, 2013 1:57:58 PM UTC-4, Zach Tellman wrote:
>>
>> I've recently been trying to pull out useful pieces from some of my more
>> monolithic libraries. The most recent result is 'byte-streams' [1], a
>> library that figures how how to convert between different byte
>> representations (including character streams), and how to efficiently
>> transfer bytes between various byte sources and sinks. The net result is
>> that you can do something like:
>>
>> (byte-streams/convert (File. "/tmp/foo") String {:encoding "utf-8"})
>>
>> and get a string representation of the file's contents. Of course, this
>> is already possible using 'slurp', but you could also convert it to a
>> CharSequence, or lazy sequence of ByteBuffers, or pretty much anything else
>> you can imagine. This is accomplished by traversing a graph of available
>> conversions (don't worry, it's memoized), so simply defining a new
>> conversion from some custom type to (say) a ByteBuffer will transitively
>> allow you to convert it to any other type.
>>
>> As an aside, this sort of conversion mechanism isn't limited to just byte
>> representations, but I'm not sure if there's another large collection of
>> mostly-isomorphic types out there that would benefit from this. If anyone
>> has ideas on where else this could be applied, I'd be interested to hear
>> them.
>>
>> Zach
>>
>> [1] https://github.com/ztellman/byte-streams
>>
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
