Hello, I am using Arrow Table's to facilitate fast data transfer between
python and R. The below strategy worked with arrow==0.14.1, but is no
longer working in arrow == 0.16.0.

Using pyarrow, I convert a pandas dataframe to a pyarrow Table, then get
the memory address to the underlying Arrow Table. Something like this:

unsigned long get_arrow_table_memory_address(py::object pyarrow_table) {
    arrow::py::import_pyarrow();
    std::shared_ptr<arrow::Table> table;
    arrow::py::unwrap_table(pyarrow_table.ptr(), &table);
    return (unsigned long) table.get();
}

Using rpy2 I can create an R process inside the python process. The arrow
table is still in memory. In the R process, I receive the memory address
(as a string, which is then converted to unsigned int in Rcpp), and return
a shared_ptr for R

SEXP arrow_table_from_memory_address(std::string memory_address) {
  std::shared_ptr<arrow::Table> table((arrow::Table *)
std::stoul(memory_address));
  Rcpp::XPtr<std::shared_ptr<arrow::Table>> output(new
std::shared_ptr<arrow::Table>(table), false);
  return output;
}

Finally, I can create a r-arrow Table object, using arrow::Table$new(xp).
My ultimate goal is to then do as.data.frame, materializing the exact same
dataframe in R as the original one in pandas.

In arrow == 0.16.0, I get an error concerning the r-arrow.so not being able
to see a symbol in libarrow_dataset.so.

10: dyn.load(file, DLLpath = DLLpath, ...)
9: library.dynam(lib, package, package.lib)
8: loadNamespace(name)
7: getNamespace(ns)
6: asNamespace(pkg)
5: get(name, envir = asNamespace(pkg), inherits = FALSE)
4: arrow:::shared_ptr at core_ArrowTablePointer.R#35
3: ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)
2: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) ||
       is.pairlist(envir)) parent.frame() else baseenv())
   .Internal(eval(expr, envir, enclos)))(expression(mydata =
ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)))
1: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) ||
       is.pairlist(envir)) parent.frame() else baseenv())
   .Internal(eval(expr, envir, enclos)))(expression(mydata =
ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE)))
Traceback (most recent call last):
  File "/root/nflx_causal_models/causal_models/r/rpy2_patches.py", line 30,
in wrapped
    return f(self, *args, **kwargs)
  File
"/opt/conda/lib/python3.7/site-packages/rpy2/rinterface_lib/conversion.py",
line 28, in _
    cdata = function(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/rpy2/rinterface.py", line
785, in __call__
    raise embedded.RRuntimeError(_rinterface._geterrmessage())
rpy2.rinterface_lib.embedded.RRuntimeError: Error in dyn.load(file, DLLpath
= DLLpath, ...) :
  unable to load shared object
'/opt/conda/lib/R/library/arrow/libs/arrow.so':
  /opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16:
undefined symbol:
_ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Running ldd on the r-arrow.so, I do see that it is properly linked against
the arrow_dataset.so

ldd /opt/conda/lib/R/library/arrow/libs/arrow.so
linux-vdso.so.1 =>  (0x00007ffc046d2000)
libarrow_dataset.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16
(0x00007ffb76a5f000)
libparquet.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libparquet.so.16
(0x00007ffb76757000)
libarrow.so.16 =>
/opt/conda/lib/R/library/arrow/libs/../../../../libarrow.so.16
(0x00007ffb757c7000)
libR.so => /opt/conda/lib/R/library/arrow/libs/../../../lib/libR.so
(0x00007ffb7532a000)


I think the symbol is hashed, so I can't tell what function in
libarrow_dataset.so it is looking for
_ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Did I need to compile a version of Arrow with some kind of flag in order to
see this symbol? I currently get arrow-cpp, pyarrow, and r-arrow all from
conda-forge.

Thank you so much for all the amazing development in arrow. This exchange
of pandas dataframe to R dataframe via arrow table is amazingly fast.
-- 
Jeffrey Wong
Computational Causal Inference

Reply via email to