I'll also add that you can write R level code that processes files
lazily chunk-by-chunk
using arrow::open_dataset() |> arrow::map_batches(), and that you can
write
C level code that processes batches chunk-by-chunk using
arrow::open_dataset() |>
nanoarrow::as_nanoarrow_array_stream(), which will give you a pointer to
an ABI stable ArrowArrayStream that you can pass between packages
(helpers
exist in the Arrow rust implementation and the nanoarrow C/C++ library
to help
consume that, and I'm happy to work up an example if you are
interested).
Cheers,
-dewey
On 2024-11-26 15:45, Bryce Mecum wrote:
The work in progress to add libarrow to MXE is located at [1]. I don't
think it's complete but I'm sharing it here in case it's a useful
starting point for someone. Continuing this work would be welcome.
Thang: Feel free to file an issue on the Arrow issue tracker [2] or
email the user@ mailing list [3] if you get stuck.
[1] https://github.com/assignUser/cran-mxe
[2] https://github.com/apache/arrow/issues
[3] https://arrow.apache.org/community/
On Tue, Nov 26, 2024 at 3:11 AM Tomas Kalibera
<tomas.kalib...@gmail.com> wrote:
On 11/26/24 11:18, Pham, T.V. via R-package-devel wrote:
> Dear all,
>
> I am planning to use libarrow for reading parquet files, chunk by chunk. It seems that
by installing "arrow", one gets the libarrow library too (at least on Windows).
On Windows, to properly use Apache Arrow with R, it would have to be
added to Rtools (and hence ideally first upstream to MXE). I've been
in
touch with two arrow developers who started working on adding libarrow
to MXE, but then ran out of time. If there were any volunteers to
finish
this, perhaps its best to check first with the arrow team where they
got, rather than starting from scratch.
Best
Tomas
>
> I wonder if the C++ header files are already in some package like Rcpp. Also,
how should the link to the libarrow look like?
>
> Thank you & best regards,
> Thang
>
> ______________________________________________________
> AmsterdamUMC disclaimer : www.amsterdamumc.org/nl/disclaimers.htm
>
> ______________________________________________
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
______________________________________________
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
