Oops -- I belatedly thought to bump up the deserialization logic to read from the same buffer that I originally wrote to and this worked fine, and clearly the problem lies elsewhere in the transit. Sorry for the noise!
Best, Jack On Mar 5, 2025 at 12:06 PM -0500, Jack Wimberley <jwimber...@paradigm4.com>, wrote: > One other bit of information: I occasionally get the error message "Tried > reading schema message, was null or length 0” instead, in the exact same > context. > On Mar 4, 2025 at 4:34 PM -0500, Jack Wimberley <jwimber...@paradigm4.com>, > wrote: > > Hello all, > > > > I am attempting to serialize and then deserialize individual RecordBatch > > objects, using the C++ library. However, I’m getting an “Invalid” result on > > the deserialization end. > > > > On the serialization end, with the help of some methods THROW_NOT_OK that > > throw on non-OK Status and Result (and returns latter case returns the > > inner Value), I’m serializing using > > > > // batch is a valid std::shared_ptr<RecordBatch> > > auto bufferStream = THROW_NOT_OK(io::BufferOutputStream::Create(1024)); > > auto batchWriter = THROW_NOT_OK(ipc::MakeStreamWriter(bufferStream, > > batch->schema())); > > auto writeStatus = THROW_NOT_OK(batchWriter->WriteRecordBatch(*batch)); > > THROW_NOT_OK(batchWriter->Close()); > > auto batchBuffer = THROW_NOT_OK(bufferStream->Finish()); > > > > // pass this data along > > > > The size of the buffer thus created is 1800. On the other end of the > > channel, I try to deserialize an in-memory copy of this IPC data using > > > > > > // bufferPtr is a uint8_t* const location in memory and bufferSize a number > > of bytes > > auto arrowBuffer = std::make_shared<Buffer>(bufferPtr, bufferSize); // > > no-copy wrap > > auto bufferReader = std::make_shared<io::BufferReader>(arrowBuffer); > > auto batchReader = > > THROW_NOT_OK(ipc::RecordBatchStreamReader::Open(bufferReader)); > > > > > > But, the last step fails, with a non-OK result with message > > > > Invalid: Expected to read 165847040 metadata bytes, but only read 1796 > > > > > > The metadata bytes size is way off, given the serialized RecordBatch was > > 1800 bytes to begin with. The number of bytes read looks about right, > > modulo that difference of 4. I saw some similar questions in the archives > > and online but the issues in them tended to be that the Close() step was > > missing. Other suggestions are a mismatch in the reader/writer format; I am > > using ones that look to me to be appropriately paired IPC stream I/O > > objects. Does some sort of header need to be written to the stream before > > the RecordBatch? Or, I did not use the second overloaded WriteRecordBatch > > method that takes a metadata object as the second argument, and the message > > mentions metadata bytes; is that relevant? > > > > Best, > > > > Jack Wimberley
