hi John, I'm not sure I follow. The EOS you're referring to is part of the streaming format. It's designed to be readable using an InputStream interface that does not support seeking at all. You can see the core logic where messages are popped off the InputStream here
https://github.com/apache/arrow/blob/6f80ea4928f0d26ca175002f2e9f511962c8b012/cpp/src/arrow/ipc/message.cc#L281 If the end of the byte stream is reached, or EOS (0) is encountered, then the stream reader stops iteration. - Wes On Tue, May 21, 2019 at 4:34 PM John Muehlhausen <j...@jgm.org> wrote: > > https://arrow.apache.org/docs/format/IPC.html#file-format > > <EOS [optional]: int32> > > If this stream marker is optional in the file format, doesn't this prevent > someone from reading the file without being able to seek() it, e.g. if it > is "piped in" to a program? Or otherwise they'll have to stream in the > entire thing before they can start parsing? > > Any reason it can't be mandatory for a File? > > -John
