I would suggest you take a look at the Arrow converter in Spark (search in the codebase), or one of the other developers or users may be able to respond on the list.
On Fri, Sep 15, 2017 at 11:41 AM, Andrew Pham (BLOOMBERG/ 731 LEX) <apha...@bloomberg.net> wrote: > Thanks Wes. I'm going over the Java source code and I have a few questions: > > Do we need to define a VectorSchemaRoot every time we define an > ArrowStreamWriter? In the interest of developing a simple proof of concept, > for now I would just like to ferry a simple byte[] array onto the > OutputStream. What would be the easiest way to construct the corresponding > VectorSchemaRoot? > > I'm also assuming that DictionaryProvider is optional > > Thanks again! > > From: dev@arrow.apache.org At: 09/14/17 18:45:15To: Andrew Pham (BLOOMBERG/ > 731 LEX ) , dev@arrow.apache.org > Subject: Re: Java Examples Writing Flatbuffer in IPC Message > > 1. Are you able to write to an shared memory IPC segment as an > OutputStream in Java? If so, do that, and use either the > ArrowFileWriter or ArrowStreamWriter in Java to write to it, like > https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java#L160 > > 2. Get a handle to that IPC segment in C++ > > 3. Wrap the C++ IPC segment in an arrow::io::BufferReader > > 4. Use the functions in arrow/ipc/reader.h to reconstruct the data > structures from the BufferReader > > There are Arrow JIRAs to make #1, #2, and #3 easier out of the box, > but you must handle these details manually for now. > > On Thu, Sep 14, 2017 at 6:37 PM, Andrew Pham (BLOOMBERG/ 731 LEX) > <apha...@bloomberg.net> wrote: >> Thanks for the reply Wes. I'm just looking for a way to ferry structured >> data (like a row in a SQL table) from one process (Java) to another (C++). >> Any ideas on how to execute this? >> >> From: dev@arrow.apache.org At: 09/14/17 18:32:12To: Andrew Pham (BLOOMBERG/ >> 731 LEX ) , dev@arrow.apache.org >> Subject: Re: Java Examples Writing Flatbuffer in IPC Message >> >> hi Andrew, >> >> Are you talking about writing a stream to shared memory? There are not >> functions in the Arrow Java library to do this out of the box. This >> would be a great contribution to make to the project: >> https://issues.apache.org/jira/browse/ARROW-721 >> >> If you create an OutputStream that writes to shared memory, you can >> use the existing stream serialization tools. In C++ the functions to >> read a stream or read a single record batch are contained in >> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/reader.h. >> >> In C++ you can use the Plasma store for writing to POSIX shared >> memory. I would like to see some generic named POSIX shared memory >> tools get developed that are independent of Plasma, though: >> https://issues.apache.org/jira/browse/ARROW-1385 >> >> Thanks >> Wes >> >> >> On Thu, Sep 14, 2017 at 6:24 PM, Andrew Pham (BLOOMBERG/ 731 LEX) >> <apha...@bloomberg.net> wrote: >>> The receiver/reader would be a C++ process. >>> >>> I'm looking for simple examples that illustrate this IPC. Can anyone point >>> me to any? Thanks! >> >> > >
