Hi Wes, Thanks so much for the feedback.
On Thu, Mar 15, 2018 at 5:38 PM, Wes McKinney <wesmck...@gmail.com> wrote: > > Per ARROW-2189 -- do you get these problems if you build from source? > I'm trying to understand if this is a packaging issue or a code issue I plan to try building from source. I will update the issue if I have any news. > On Mon, Feb 26, 2018 at 10:22 AM, Rares Vernica <rvern...@gmail.com> wrote: > >> - On the coordinator side, do I really need to read and write a record >> batch? Could I copy the buffer directly somehow? > > No, you don't need to necessarily. The idea of the Message* classes in > arrow::ipc is to facilitate transporting messages while being agnostic > to their comments. This would be a useful test case to flesh out these > APIs. Could you please open some JIRAs about this? There are already > some Message-related JIRAs open so take a look at what is there > already. I found these JIRA issues related to this: * https://issues.apache.org/jira/browse/ARROW-1860 * https://issues.apache.org/jira/browse/ARROW-2006 * https://issues.apache.org/jira/browse/ARROW-2027 For my case, would the following work: 0. Get the uint8_t* "data" and uint32_t "size" pair from my underlying framework 1. Create a Buffer Reader using the "data" and "size" pair 2. Create a Message Reader using Buffer Reader 3. While there are messages in the Message Reader 3.1. Read next Message 3.2. Write Message to Output Stream All of this will happen at the coordinator, so it will loop over the nodes and process all the "data" received from each in no particular order. Nevertheless, all the data of one node is processed at once entirely. Cheers, Rares
