Yes definitely! You can do this through high level Python APIs, e.g.,
something like
https://github.com/apache/arrow/blob/ca3acdc138b1ac27c9111b236d33593988689a20/python/pyarrow/tests/test_serialization.py#L214-L216
.

You can also share the numpy arrays using shared memory, e.g.,
https://issues.apache.org/jira/browse/ARROW-1792?focusedCommentId=16252940&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16252940

You can also do this through C++.

Some benchmarks at
https://ray-project.github.io/2017/10/15/fast-python-serialization-with-ray-and-arrow.html
.
On Thu, Nov 16, 2017 at 10:49 AM Lewis John McGibbney <lewi...@apache.org>
wrote:

> Hi Folks,
>
> Array-oriented scientific data (such as satellite remote sensing data) is
> commonly encoded using NetCDF [0] and HDF [1] data formats as these formats
> have been designed and developed to offer amongst other things, some/all of
> the following features
>  * Self-Describing. A netCDF file includes information about the data it
> contains.
> * Portable. A netCDF file can be accessed by computers with different ways
> of storing integers, characters, and floating-point numbers.
>  * Scalable. A small subset of a large dataset may be accessed efficiently.
>  * Appendable. Data may be appended to a properly structured netCDF file
> without copying the dataset or redefining its structure.
>  * Sharable. One writer and multiple readers may simultaneously access the
> same netCDF file.
>  * Archivable. Access to all earlier forms of netCDF data will be
> supported by current and future versions of the software.
>
> I am currently toying with the idea of exploring and hopefully
> benchmarking use of storage-class memory hardware combined with Arrow as a
> mechanism for improving both fast and flexible data access and possibly
> analysis.
>
> Very first question, has anyone attempted to/are currently using Arrow to
> store N-Dim array-based data?
>
> Thanks in advance,
> Lewis
>
> [0] http://www.unidata.ucar.edu/software/netcdf/
> [1] https://www.hdfgroup.org/solutions/hdf5/
>

Reply via email to