Yes definitely! You can do this through high level Python APIs, e.g., something like https://github.com/apache/arrow/blob/ca3acdc138b1ac27c9111b236d33593988689a20/python/pyarrow/tests/test_serialization.py#L214-L216 .
You can also share the numpy arrays using shared memory, e.g., https://issues.apache.org/jira/browse/ARROW-1792?focusedCommentId=16252940&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16252940 You can also do this through C++. Some benchmarks at https://ray-project.github.io/2017/10/15/fast-python-serialization-with-ray-and-arrow.html . On Thu, Nov 16, 2017 at 10:49 AM Lewis John McGibbney <lewi...@apache.org> wrote: > Hi Folks, > > Array-oriented scientific data (such as satellite remote sensing data) is > commonly encoded using NetCDF [0] and HDF [1] data formats as these formats > have been designed and developed to offer amongst other things, some/all of > the following features > * Self-Describing. A netCDF file includes information about the data it > contains. > * Portable. A netCDF file can be accessed by computers with different ways > of storing integers, characters, and floating-point numbers. > * Scalable. A small subset of a large dataset may be accessed efficiently. > * Appendable. Data may be appended to a properly structured netCDF file > without copying the dataset or redefining its structure. > * Sharable. One writer and multiple readers may simultaneously access the > same netCDF file. > * Archivable. Access to all earlier forms of netCDF data will be > supported by current and future versions of the software. > > I am currently toying with the idea of exploring and hopefully > benchmarking use of storage-class memory hardware combined with Arrow as a > mechanism for improving both fast and flexible data access and possibly > analysis. > > Very first question, has anyone attempted to/are currently using Arrow to > store N-Dim array-based data? > > Thanks in advance, > Lewis > > [0] http://www.unidata.ucar.edu/software/netcdf/ > [1] https://www.hdfgroup.org/solutions/hdf5/ >