Fantastic Robert, thank you for the pointers. The documentation and graphics on ray github pages is very helpful. Lewis
On 2017-11-16 11:20, Robert Nishihara <robertnishih...@gmail.com> wrote: > Yes definitely! You can do this through high level Python APIs, e.g., > something like > https://github.com/apache/arrow/blob/ca3acdc138b1ac27c9111b236d33593988689a20/python/pyarrow/tests/test_serialization.py#L214-L216 > . > > You can also share the numpy arrays using shared memory, e.g., > https://issues.apache.org/jira/browse/ARROW-1792?focusedCommentId=16252940&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16252940 > > You can also do this through C++. > > Some benchmarks at > https://ray-project.github.io/2017/10/15/fast-python-serialization-with-ray-and-arrow.html > . > On Thu, Nov 16, 2017 at 10:49 AM Lewis John McGibbney <lewi...@apache.org> > wrote: > > > Hi Folks, > > > > Array-oriented scientific data (such as satellite remote sensing data) is > > commonly encoded using NetCDF [0] and HDF [1] data formats as these formats > > have been designed and developed to offer amongst other things, some/all of > > the following features > > * Self-Describing. A netCDF file includes information about the data it > > contains. > > * Portable. A netCDF file can be accessed by computers with different ways > > of storing integers, characters, and floating-point numbers. > > * Scalable. A small subset of a large dataset may be accessed efficiently. > > * Appendable. Data may be appended to a properly structured netCDF file > > without copying the dataset or redefining its structure. > > * Sharable. One writer and multiple readers may simultaneously access the > > same netCDF file. > > * Archivable. Access to all earlier forms of netCDF data will be > > supported by current and future versions of the software. > > > > I am currently toying with the idea of exploring and hopefully > > benchmarking use of storage-class memory hardware combined with Arrow as a > > mechanism for improving both fast and flexible data access and possibly > > analysis. > > > > Very first question, has anyone attempted to/are currently using Arrow to > > store N-Dim array-based data? > > > > Thanks in advance, > > Lewis > > > > [0] http://www.unidata.ucar.edu/software/netcdf/ > > [1] https://www.hdfgroup.org/solutions/hdf5/ > > >
