Re: C++ RecordBatchWriter/ReadRecordBatch clarification

2018-04-23 Thread Wes McKinney
> So, I guess the ReadRecordBatch function is intended to only work if the > records were written by RecordBatchFileWriter, right? The *StreamWriter and *FileWriter classes use identical code paths for writing the IPC messages, the only difference is the preamble (the magic number and padding) an

Re: C++ RecordBatchWriter/ReadRecordBatch clarification

2018-04-22 Thread Rares Vernica
Hi Dimitri, Thanks for that. I was going something like that. My hope was to use StreamWriter to write the batch and then use ReadRecordBatch to read it since it is more succinct and I know I only have one batch to read. Here is the actual code I use https://github.com/Paradigm4/accelerated_io_to

Re: C++ RecordBatchWriter/ReadRecordBatch clarification

2018-04-17 Thread Dimitri Vorona
Hi Rares, you use a different reader for the RecordBatch streams. See arrow/ipc/ipc-read-write-test.cc:569-596 for the gist. Also, the second argument to arrow::RecordBatch::Make takes the number of rows in the batch, so you have to set it to 1 in your example. See https://gist.github.com/alendi

C++ RecordBatchWriter/ReadRecordBatch clarification

2018-04-16 Thread Rares Vernica
Hi, I'm writing a batch of records to a stream and I want to read them later. I notice that if I use the RecordBatchStreamWriter class to write them and then ReadRecordBatch function to read them, I get a Segmentation Fault. On the other hand, if I use the RecordBatchFileWriter class to write the