Domink's point is that the IPC reader currently first writes the whole
thing into a Vec, and then copies all of that to buffers using
IPC::Buffer offsets and lengths. Thus, it performs 2 memcopies of the whole
data and needs to hold 2x the required memory (the Vec and the
arrow::Buffers).
I notice
Thank you for filing the ticket.
I wonder if you mean this reader:
https://docs.rs/arrow/3.0.0/arrow/ipc/reader/struct.FileReader.html#method.try_new
If so, while it is called a `FileReader` I think that is somewhat
misleading. It requires something that implements `std::io::Read` -- which
`&[u8]
I just remembered a bigger issue I ran into. I wanted to read from IPC but
I don’t have a file. I do have the data as [u8] already. The current API
incurs more copies than necessary (I think) and therefore the performance
of reading IPC is worse than in JS. (
https://issues.apache.org/jira/project
I am looking forward to speaking with you then. I’ll talk about the
motivation.
My experience with the library has been good. I ran into a few limitations
that I filed Jiras for. I struggled a bit with some of the error handling
and Arc types but that’s probably because I am now very experienced
Hi Dominik,
That sounds really interesting -- thank you for the offer
I for one would enjoy seeing a demo and suggest that 10 minutes might be a
good length. The next call (details are also on the announcement [1]) is
scheduled for Wednesday March 10, 2021 at 09:00 PST / 12:00 EST / 17:00
UTC. T
Hi Dominic,
I would be interested in a demo. Im curious to see your implementation and
what advantages you have seen over javascript
thanks
Fernando
On Thu, Feb 25, 2021 at 10:39 PM Dominik Moritz wrote:
> Hello Rust Arrow Devs,
>
> I have been working on a wasm version of Arrow using the Rust
