Re: Modeling N-Dim Arrays in Arrow

2017-11-16 Thread Robert Nishihara
Great! On Thu, Nov 16, 2017 at 3:04 PM Lewis John McGibbney wrote: > Fantastic Robert, thank you for the pointers. > The documentation and graphics on ray github pages is very helpful. > Lewis > > On 2017-11-16 11:20, Robert Nishihara wrote: > > Yes definitely! You can do this through high leve

Re: Modeling N-Dim Arrays in Arrow

2017-11-16 Thread Lewis John McGibbney
Fantastic Robert, thank you for the pointers. The documentation and graphics on ray github pages is very helpful. Lewis On 2017-11-16 11:20, Robert Nishihara wrote: > Yes definitely! You can do this through high level Python APIs, e.g., > something like > https://github.com/apache/arrow/blob/ca3

Re: Modeling N-Dim Arrays in Arrow

2017-11-16 Thread Robert Nishihara
Yes definitely! You can do this through high level Python APIs, e.g., something like https://github.com/apache/arrow/blob/ca3acdc138b1ac27c9111b236d33593988689a20/python/pyarrow/tests/test_serialization.py#L214-L216 . You can also share the numpy arrays using shared memory, e.g., https://issues.ap

Modeling N-Dim Arrays in Arrow

2017-11-16 Thread Lewis John McGibbney
Hi Folks, Array-oriented scientific data (such as satellite remote sensing data) is commonly encoded using NetCDF [0] and HDF [1] data formats as these formats have been designed and developed to offer amongst other things, some/all of the following features * Self-Describing. A netCDF file in