I suggest that you use the Arrow streaming format for this purpose instead (http://arrow.apache.org/docs/python/ipc.html). The Feather APIs exist for backwards compatibility, and require an IO interface that supports random access.
- Wes On Sun, Sep 10, 2017 at 10:05 PM, Rares Vernica <[email protected]> wrote: > Hi, > > I am having trouble piping Feather structures between two processes. On the > receiving-process side, I get: pyarrow.lib.ArrowIOError: [Errno 29] Illegal > seek > > > I have process A and process B which communicate via pipes. Process A sends > the bytes of a Feather structure to process B. Process A could send one or > more Feather structures, but, after each one, it waits for a response from > B. The pipe is to remain open, so there is no EOF after each Feather > structure. > > The receiving process is a Python process and I do something like: > > pyarrow.feather.read_feather(sys.stdin) > > The error I receive it: > > File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 143, > in read_feather > reader = FeatherReader(source) > File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 43, > in __init__ > self.open(source) > File "pyarrow/feather.pxi", line 83, in pyarrow.lib.FeatherReader.open > (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:60120) > File "pyarrow/error.pxi", line 72, in pyarrow.lib.check_status > (/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7495) > pyarrow.lib.ArrowIOError: [Errno 29] Illegal seek > > If instead of process A, I redirect a file, it works fine: > > # python -uc "import pyarrow.feather, sys; > print(pyarrow.feather.read_feather(sys.stdin))" < a.feather > id > 0 1.0 > 1 2.0 > 2 3.0 > 3 NaN > 4 5.0 > 5 6.0 > 6 7.0 > 7 8.0 > > The difference between process A and redirecting a file is that process A > does not send EOF or close the pipe. Does read_feather need EOF? > > I also tried reading the bytes from process A in memory, writing them to a > file, and then reading the file with read_feather. This works fine. So, I > believe process A sends a complete Feather structure. > > Any thoughts? Thanks! > -- > Rares
