Agreed -- please check 0.17.0. Also, note that the C Data Interface makes moving shared_ptr<T> between R and Python radically simpler. See the "py_to_r" functions in
https://github.com/apache/arrow/blob/master/r/tests/testthat/test-python.R Similar "r_to_py" functions could be written to use rpy2 If you can provide instructions for reproducing this error in an isolated environment that would be helpful On Sun, Apr 26, 2020 at 5:44 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > > Hi Jeffrey, > I don't have expertise in this area (hopefully someone else can chime in), > but we recently released 0.17.0 could you check if this is still an issue > with the newer version? > > Thanks, > Micah > > On Sat, Apr 25, 2020 at 10:43 PM Jeffrey Wong <jeffr...@netflix.com.invalid> > wrote: > > > I was able to simplify this very much. There is a problem with > > pyarrow==0.16.0, r-arrow==0.16.0, and rpy2. Just by loading pyarrow, rpy2 > > will not be able to load r-arrow. This set of imports fails now, but was > > fine in 0.14.1. Is it possible there is a conflict with shared objects that > > pyarrow loads, and shared objects that r-arrow tries to load after? > > > > # Fails > > import rpy2.robjects as ro > > import pyarrow > > ro.r("library(arrow)") > > > > # Succeeds > > import rpy2.robjects as ro > > ro.r("library(arrow)") > > > > # Also fails > > import rpy2.robjects as ro > > import pyarrow > > import pyarrow.parquet > > import pyarrow.dataset > > ro.r("library(arrow)") > > > > On Sat, Apr 25, 2020 at 12:19 PM Jeffrey Wong <jeffr...@netflix.com> > > wrote: > > > > > Hello, I am using Arrow Table's to facilitate fast data transfer between > > > python and R. The below strategy worked with arrow==0.14.1, but is no > > > longer working in arrow == 0.16.0. > > > > > > Using pyarrow, I convert a pandas dataframe to a pyarrow Table, then get > > > the memory address to the underlying Arrow Table. Something like this: > > > > > > unsigned long get_arrow_table_memory_address(py::object pyarrow_table) { > > > arrow::py::import_pyarrow(); > > > std::shared_ptr<arrow::Table> table; > > > arrow::py::unwrap_table(pyarrow_table.ptr(), &table); > > > return (unsigned long) table.get(); > > > } > > > > > > Using rpy2 I can create an R process inside the python process. The arrow > > > table is still in memory. In the R process, I receive the memory address > > > (as a string, which is then converted to unsigned int in Rcpp), and > > return > > > a shared_ptr for R > > > > > > SEXP arrow_table_from_memory_address(std::string memory_address) { > > > std::shared_ptr<arrow::Table> table((arrow::Table *) > > > std::stoul(memory_address)); > > > Rcpp::XPtr<std::shared_ptr<arrow::Table>> output(new > > > std::shared_ptr<arrow::Table>(table), false); > > > return output; > > > } > > > > > > Finally, I can create a r-arrow Table object, using arrow::Table$new(xp). > > > My ultimate goal is to then do as.data.frame, materializing the exact > > same > > > dataframe in R as the original one in pandas. > > > > > > In arrow == 0.16.0, I get an error concerning the r-arrow.so not being > > > able to see a symbol in libarrow_dataset.so. > > > > > > 10: dyn.load(file, DLLpath = DLLpath, ...) > > > 9: library.dynam(lib, package, package.lib) > > > 8: loadNamespace(name) > > > 7: getNamespace(ns) > > > 6: asNamespace(pkg) > > > 5: get(name, envir = asNamespace(pkg), inherits = FALSE) > > > 4: arrow:::shared_ptr at core_ArrowTablePointer.R#35 > > > 3: ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE) > > > 2: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) > > || > > > is.pairlist(envir)) parent.frame() else baseenv()) > > > .Internal(eval(expr, envir, enclos)))(expression(mydata = > > > ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE))) > > > 1: (function (expr, envir = parent.frame(), enclos = if (is.list(envir) > > || > > > is.pairlist(envir)) parent.frame() else baseenv()) > > > .Internal(eval(expr, envir, enclos)))(expression(mydata = > > > ArrowTablePointer$new("94637300534352")$to_table(as_tibble = FALSE))) > > > Traceback (most recent call last): > > > File "/root/nflx_causal_models/causal_models/r/rpy2_patches.py", line > > > 30, in wrapped > > > return f(self, *args, **kwargs) > > > File > > > > > "/opt/conda/lib/python3.7/site-packages/rpy2/rinterface_lib/conversion.py", > > > line 28, in _ > > > cdata = function(*args, **kwargs) > > > File "/opt/conda/lib/python3.7/site-packages/rpy2/rinterface.py", line > > > 785, in __call__ > > > raise embedded.RRuntimeError(_rinterface._geterrmessage()) > > > rpy2.rinterface_lib.embedded.RRuntimeError: Error in dyn.load(file, > > > DLLpath = DLLpath, ...) : > > > unable to load shared object > > > '/opt/conda/lib/R/library/arrow/libs/arrow.so': > > > /opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16: > > > undefined symbol: > > > > > _ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE > > > > > > Running ldd on the r-arrow.so, I do see that it is properly linked > > against > > > the arrow_dataset.so > > > > > > ldd /opt/conda/lib/R/library/arrow/libs/arrow.so > > > linux-vdso.so.1 => (0x00007ffc046d2000) > > > libarrow_dataset.so.16 => > > > /opt/conda/lib/R/library/arrow/libs/../../../../libarrow_dataset.so.16 > > > (0x00007ffb76a5f000) > > > libparquet.so.16 => > > > /opt/conda/lib/R/library/arrow/libs/../../../../libparquet.so.16 > > > (0x00007ffb76757000) > > > libarrow.so.16 => > > > /opt/conda/lib/R/library/arrow/libs/../../../../libarrow.so.16 > > > (0x00007ffb757c7000) > > > libR.so => /opt/conda/lib/R/library/arrow/libs/../../../lib/libR.so > > > (0x00007ffb7532a000) > > > > > > > > > I think the symbol is hashed, so I can't tell what function in > > > libarrow_dataset.so it is looking for > > > > > > > > _ZN5arrow2fs8internal17SplitAbstractPathERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE > > > > > > Did I need to compile a version of Arrow with some kind of flag in order > > > to see this symbol? I currently get arrow-cpp, pyarrow, and r-arrow all > > > from conda-forge. > > > > > > Thank you so much for all the amazing development in arrow. This exchange > > > of pandas dataframe to R dataframe via arrow table is amazingly fast. > > > -- > > > Jeffrey Wong > > > Computational Causal Inference > > > > > > > > > -- > > Jeffrey Wong > > Computational Causal Inference > >
