Hi, Isaiah. This is a valid question. 0. As a preface, I'd like to say I'm not trying to replace anything. I wrote RawArray to solve a problem we have in magnetic resonance imaging (quickly saving and loading large complex float arrays), and then I decided to share it so if other people like it and find it useful, then cool beans.
Now for the mild stumping... 1. I don't think NRRD is as substantially used as you might think. I've worked in imaging science for years on the data processing/file format end, and I've never seen anyone use it, and I've never even heard of it. (Pity, because it looks nice enough. :-\) 2. RawArray is simpler to handle and trivial to understand. I believe all you need from an I/O library is I/O.* I don't want my file I/O library performing transformations on my data. I also don't need it to read image formats. Part of the reason behind RawArray is to avoid standard image formats because they are not optimized for large complex-float arrays. I just want to save multi-GB data arrays to disk quickly and read them back quickly on a different machine, five years later. I have other implementations (https://github.com/davidssmith/ra), and all are super short and platform agnostic. 3. RawArray is surely faster. All it does is read. It doesn't perform any transformations or encoding, so it can't possibly be slower than NRRD. There is a C library at (https://github.com/davidssmith/ra) if you think a pure Julia implementation isn't fast enough. Cheers, Dave [*] That said, I'm not completely ruling out having transformations available in RawArray between the RAM and disk. For example, when I first wrote it, I had included Blosc compression as an option, signaled by a flag in the header. But in general most transformations are best made in RAM after reading or on disk with already existing, battle-proven tools, such as gzip, uunencode, tar, etc. On Sunday, September 25, 2016 at 9:59:45 PM UTC-5, Isaiah wrote: > > Is there a reason to use this file format over NRRD [1]? To borrow a wise > phrasing: I wonder if the world needs another lightweight raw data format ;) > > For what it's worth, NRRD is already supported by JuliaIO/Images.jl, and I > believe addresses the use-cases identified in your readme, but with a > number of technical and non-technical advantages (not least: a number of > independent implementations, and a substantial user base, at least as far > as these things go). > > I say this -- very selfishly I admit -- as someone who has been on the > receiving end of far too many files in home-brewed formats. > > [1] http://teem.sourceforge.net/nrrd/descformat.html > > On Sunday, September 25, 2016, David Smith <[email protected] > <javascript:>> wrote: > >> Hi, all: >> >> I finally pushed this out, and it might satisfy some of your needs for a >> simple way to store N-d arrays to disk. Hope you enjoy it. >> >> RawArray (.ra) is a simple file format for storing n-dimensional arrays. >> RawArray was designed to be portable, fast, storage efficient, and future >> proof. Basically it writes the binary array data directly to disk with a >> short header that is used to recreate type and dimension information. >> >> RawArray is faster than HDF5 and supports complex numbers out of the box, >> which HDF5 does not. RawArray supports all basic `Int`, `UInt`, `Float`, >> and `Complex{}` types, and more can be easily added in the future, such as >> Rational or Big*. It can also handle derived types, but the serialization >> of them is currently left up to the user. >> >> A system of version numbers and flags are implemented to future-proof the >> data files as well, in case the implementation needs to change for some >> reason. >> >> You can grab it with `Pkg.add("RawArray")`. A minimum of Julia 0.4 is >> required. >> >> Repository: https://github.com/davidssmith/RawArray.jl >> >> Cheers, >> Dave >> >
