Sorry I forgot to add: JuliaIO/Images.jl relies on having ImageMagick installed, whereas RawArray.jl is a pure Julia solution without any dependencies.
On Monday, September 26, 2016 at 8:59:15 AM UTC-5, David Smith wrote: > > Hi, Isaiah. This is a valid question. > > 0. As a preface, I'd like to say I'm not trying to replace anything. I > wrote RawArray to solve a problem we have in magnetic resonance imaging > (quickly saving and loading large complex float arrays), and then I decided > to share it so if other people like it and find it useful, then cool beans. > > Now for the mild stumping... > > 1. I don't think NRRD is as substantially used as you might think. I've > worked in imaging science for years on the data processing/file format end, > and I've never seen anyone use it, and I've never even heard of it. (Pity, > because it looks nice enough. :-\) > > 2. RawArray is simpler to handle and trivial to understand. I believe all > you need from an I/O library is I/O.* I don't want my file I/O library > performing transformations on my data. > > I also don't need it to read image formats. Part of the reason behind > RawArray is to avoid standard image formats because they are not optimized > for large complex-float arrays. I just want to save multi-GB data arrays to > disk quickly and read them back quickly on a different machine, five years > later. > > I have other implementations (https://github.com/davidssmith/ra), and all > are super short and platform agnostic. > > 3. RawArray is surely faster. All it does is read. It doesn't perform any > transformations or encoding, so it can't possibly be slower than NRRD. > There is a C library at (https://github.com/davidssmith/ra) if you think > a pure Julia implementation isn't fast enough. > > Cheers, > Dave > > [*] That said, I'm not completely ruling out having transformations > available in RawArray between the RAM and disk. For example, when I first > wrote it, I had included Blosc compression as an option, signaled by a flag > in the header. But in general most transformations are best made in RAM > after reading or on disk with already existing, battle-proven tools, such > as gzip, uunencode, tar, etc. > > > On Sunday, September 25, 2016 at 9:59:45 PM UTC-5, Isaiah wrote: >> >> Is there a reason to use this file format over NRRD [1]? To borrow a wise >> phrasing: I wonder if the world needs another lightweight raw data format ;) >> >> For what it's worth, NRRD is already supported by JuliaIO/Images.jl, and >> I believe addresses the use-cases identified in your readme, but with a >> number of technical and non-technical advantages (not least: a number of >> independent implementations, and a substantial user base, at least as far >> as these things go). >> >> I say this -- very selfishly I admit -- as someone who has been on the >> receiving end of far too many files in home-brewed formats. >> >> [1] http://teem.sourceforge.net/nrrd/descformat.html >> >> On Sunday, September 25, 2016, David Smith <[email protected]> wrote: >> >>> Hi, all: >>> >>> I finally pushed this out, and it might satisfy some of your needs for a >>> simple way to store N-d arrays to disk. Hope you enjoy it. >>> >>> RawArray (.ra) is a simple file format for storing n-dimensional arrays. >>> RawArray was designed to be portable, fast, storage efficient, and future >>> proof. Basically it writes the binary array data directly to disk with a >>> short header that is used to recreate type and dimension information. >>> >>> RawArray is faster than HDF5 and supports complex numbers out of the >>> box, which HDF5 does not. RawArray supports all basic `Int`, `UInt`, >>> `Float`, and `Complex{}` types, and more can be easily added in the future, >>> such as Rational or Big*. It can also handle derived types, but the >>> serialization of them is currently left up to the user. >>> >>> A system of version numbers and flags are implemented to future-proof >>> the data files as well, in case the implementation needs to change for some >>> reason. >>> >>> You can grab it with `Pkg.add("RawArray")`. A minimum of Julia 0.4 is >>> required. >>> >>> Repository: https://github.com/davidssmith/RawArray.jl >>> >>> Cheers, >>> Dave >>> >>
