Hello, I am using Arrow Table's to facilitate fast data transfer between
python and R. The below strategy worked with arrow==0.14.1, but is no
longer working in arrow == 0.16.0.
Using pyarrow, I convert a pandas dataframe to a pyarrow Table, then get
the memory address to the underlying Arrow Table. Something like this:
unsigned long get_arrow_table_memory_address(py::object pyarrow_table) {
arrow::py::import_pyarrow();
std::shared_ptr<arrow::Table> table;
arrow::py::unwrap_table(pyarrow_table.ptr(), &table);
return (unsigned long) table.get();
}
Using rpy2 I can create an R process inside the python process. The arrow
table is still in memory. In the R process, I receive the memory address
(as a string, which is then converted to unsigned int in Rcpp), and return
a shared_ptr for R
SEXP arrow_table_from_memory_address(std::string memory_address) {
std::shared_ptr<arrow::Table> table((arrow::Table *)
std::stoul(memory_address));
Rcpp::XPtr<std::shared_ptr<arrow::Table>> output(new
std::shared_ptr<arrow::Table>(table), false);
return output;
}
Finally, I can create a r-arrow Table object, using arrow::Table$new(xp).
My ultimate goal is to then do as.data.frame, materializing the exact same
dataframe in R as the original one in pandas.
In arrow == 0.16.0, I get an error concerning the r-arrow.so not being able
to see a symbol in libarrow_dataset.so.
10: dyn.load(file, DLLpath = DLLpath, ...)
9: library.dynam(lib, package, package.lib)
8: loadNamespace(name)
7: getNamespace(ns)
6: asNamespace(pkg)
5: get(name, envir = asNamespace(pkg), inherits = FALSE)
4: arrow:::shared_ptr at core_ArrowTablePointer.R#35
3: ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)
2: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) ||
is.pairlist(envir)) parent.frame() else baseenv())
.Internal(eval(expr, envir, enclos)))(expression(mydata =
ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)))
1: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) ||
is.pairlist(envir)) parent.frame() else baseenv())
.Internal(eval(expr, envir, enclos)))(expression(mydata =
ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)))
Traceback (most recent call last):
File "/root/nflx_causal_models/causal_models/r/rpy2_patches.py", line 30,
in wrapped
return f(self, *args, **kwargs)
File
"/opt/conda/lib/python3.7/site-packages/rpy2/rinterface_lib/conversion.py",
line 28, in _
cdata = function(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/rpy2/rinterface.py", line
785, in __call__
raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error in dyn.load(file, DLLpath
= DLLpath, ...) :
unable to load shared object
'/opt/conda/lib/R/library/arrow/libs/arrow.so':
/opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16:
undefined symbol:
_ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Running ldd on the r-arrow.so, I do see that it is properly linked against
the arrow_dataset.so
ldd /opt/conda/lib/R/library/arrow/libs/arrow.so
linux-vdso.so.1 => (0x00007ffc046d2000)
libarrow_dataset.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16
(0x00007ffb76a5f000)
libparquet.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libparquet.so.16
(0x00007ffb76757000)
libarrow.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libarrow.so.16
(0x00007ffb757c7000)
libR.so => /opt/conda/lib/R/library/arrow/libs/../../../lib/libR.so
(0x00007ffb7532a000)
I think the symbol is hashed, so I can't tell what function in
libarrow_dataset.so it is looking for
_ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Did I need to compile a version of Arrow with some kind of flag in order to
see this symbol? I currently get arrow-cpp, pyarrow, and r-arrow all from
conda-forge.
Thank you so much for all the amazing development in arrow. This exchange
of pandas dataframe to R dataframe via arrow table is amazingly fast.
--
Jeffrey Wong
Computational Causal Inference
