Hi,
I am having trouble piping Feather structures between two processes. On the
receiving-process side, I get: pyarrow.lib.ArrowIOError: [Errno 29] Illegal
seek
I have process A and process B which communicate via pipes. Process A sends
the bytes of a Feather structure to process B. Process A could send one or
more Feather structures, but, after each one, it waits for a response from
B. The pipe is to remain open, so there is no EOF after each Feather
structure.
The receiving process is a Python process and I do something like:
pyarrow.feather.read_feather(sys.stdin)
The error I receive it:
File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 143,
in read_feather
reader = FeatherReader(source)
File "/usr/local/lib/python2.7/dist-packages/pyarrow/feather.py", line 43,
in __init__
self.open(source)
File "pyarrow/feather.pxi", line 83, in pyarrow.lib.FeatherReader.open
(/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:60120)
File "pyarrow/error.pxi", line 72, in pyarrow.lib.check_status
(/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:7495)
pyarrow.lib.ArrowIOError: [Errno 29] Illegal seek
If instead of process A, I redirect a file, it works fine:
# python -uc "import pyarrow.feather, sys;
print(pyarrow.feather.read_feather(sys.stdin))" < a.feather
id
0 1.0
1 2.0
2 3.0
3 NaN
4 5.0
5 6.0
6 7.0
7 8.0
The difference between process A and redirecting a file is that process A
does not send EOF or close the pipe. Does read_feather need EOF?
I also tried reading the bytes from process A in memory, writing them to a
file, and then reading the file with read_feather. This works fine. So, I
believe process A sends a complete Feather structure.
Any thoughts? Thanks!
--
Rares
