hi Matt, Arrow in Go is still very much WIP territory AFAICT. I've been hoping to see more of a developer community grow here but we still have a ways to go it seems. You can help get one going =)
On Wed, Oct 10, 2018 at 2:53 PM Matthew Perpick <ma...@datadoghq.com> wrote: > > I'm experimenting with arrow in an online streaming analytics project. It > looks like it could potentially be a great fit but i'm hitting a few > roadblocks. > > A goal for my project is to stream data into pre-allocated arrays in memory > and include them in analytics with arrays stored elsewhere (db, disk, etc). > I'm currently hitting a barrier with Array Builders. They can't produce an > array without resetting their memory (of course, correct me if I'm wrong). > This forces me to couple the sizes of my buffers in memory/disk with the > query rate rather than something more logical like time elapsed, bytes, > number of elements. > > Ultimately, I'd like to have array builders be able to produce an unsafe > array which shares the same memory buffers. I'd have to protect > resizes/copies/resets with mutexes, but I believe it should work. Is this a > case you'd like to support with Arrow? I have an incomplete PR > <https://github.com/clutchski/arrow/pull/1/files>which illustrates the > idea, which I could complete if you all think it's a reasonable direction. If you could open a JIRA issue and describe the change you are proposing to make, it would be then good to go ahead and open a PR to stir the pot. Some of the existing contributors to the Go library will be able to comment > > One other question: The Go implementation seems pretty incomplete. If i > wanted to better understand Arrow's full capabilities, which implementation > is most complete? C++? The deepest library is indeed the C++ library, so that would be a good place to start. - Wes > > Thanks for all the work so far. > Matt
