bparks13 opened a new issue, #409: URL: https://github.com/apache/arrow-dotnet/issues/409
### Describe the bug, including details regarding any error messages, version, and platform. ## Environment - `Apache.Arrow` / `Apache.Arrow.Compression` 23.0.0, `net472`, x64, Windows 11 - `ArrowFileWriter` over a plain `FileStream`, writing record batches continuously for the duration of a recording ## Background We are using `Apache.Arrow` to save continuous streaming data from [ONIX hardware](https://open-ephys.github.io/onix-docs/), where our project ([OpenEphys.Onix1](https://github.com/open-ephys/bonsai-onix1)) is hosted as a package that can run in [Bonsai-Rx](https://github.com/bonsai-rx/bonsai). ## Problem Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its `Block` offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises `ArrowInvalid: Unexpected empty message in IPC file format`. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter. `ArrowFileWriter` takes the block offset from `BaseStream.Position` and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and `FileStream` having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental. For more details, we are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems. ## Our usage - [`ArrowWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowWriter.cs) — construction, `WriteRecordBatch`, and `WriteEnd` on dispose - [`ArrowBatchWriter.cs`](https://github.com/open-ephys/bonsai-onix1/blob/d5b8370/OpenEphys.Onix1/DataFrameWriter/ArrowBatchWriter.cs) — batches are accumulated with an Rx `Buffer(timeout, count)`, so `WriteRecordBatch` calls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush) Note also that our `ArrowBuffer`s wrap unmanaged memory via a custom `MemoryManager<byte>` to handle `OpenCV.Net.Mat` objects. I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know. ## Questions 1. Is there any path in `ArrowFileWriter` that advances `BaseStream.Position` without writing bytes (a seek, or padding emitted by seeking)? 2. Is there anything we should be doing differently? `FileOptions.WriteThrough`, periodic `Flush(true)` on the base stream, or otherwise to keep a lower layer from silently dropping a write like this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
