Hi Folks, Array-oriented scientific data (such as satellite remote sensing data) is commonly encoded using NetCDF [0] and HDF [1] data formats as these formats have been designed and developed to offer amongst other things, some/all of the following features * Self-Describing. A netCDF file includes information about the data it contains. * Portable. A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers. * Scalable. A small subset of a large dataset may be accessed efficiently. * Appendable. Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure. * Sharable. One writer and multiple readers may simultaneously access the same netCDF file. * Archivable. Access to all earlier forms of netCDF data will be supported by current and future versions of the software.
I am currently toying with the idea of exploring and hopefully benchmarking use of storage-class memory hardware combined with Arrow as a mechanism for improving both fast and flexible data access and possibly analysis. Very first question, has anyone attempted to/are currently using Arrow to store N-Dim array-based data? Thanks in advance, Lewis [0] http://www.unidata.ucar.edu/software/netcdf/ [1] https://www.hdfgroup.org/solutions/hdf5/
