How about using JLD.jl: https://github.com/JuliaIO/JLD.jl
On Tuesday, September 13, 2016 at 7:00:25 PM UTC+2, sparrowhawker wrote: > > Hi, > > I'm new to Julia, and have been able to accomplish a lot of what I used to > do in Matlab/Fortran, in very little time since I started using Julia in > the last three months. Here's my newest stumbling block. > > I have a process which creates nsamples within a loop. Each sample takes a > long time to compute as there are expensive finite difference operations, > which ultimately lead to a sample, say 1 to 10 seconds. I have to store > each of the nsamples, and I know the size and dimensions of each of the > nsamples (all samples have the same size and dimensions). However, > depending on the run time parameters, each sample may be a 32x32 image or > perhaps a 64x64x64 voxset with 3 attributes, i.e., a 64x64x64x3 > hyper-rectangle. To be clear, each sample can be an arbitrary dimension > hyper-rectangle, specified at run time. > > Obviously, since I don't want to lose computation and want to see > incremental progress, I'd like to do incremental saves of these samples on > disk, instead of waiting to collect all nsamples at the end. For instance, > if I had to store 1000 samples of size 64x64, I thought perhaps I could > chunk and save 64x64 slices to an HDF5 file 1000 times. Is this the right > approach? If so, here's a prototype program to do so, but it depends on my > knowing the number of dimensions of the slice, which is not known until > runtime, > > using HDF5 > > filename = "test.h5" > # open file > fmode ="w" > # get a file object > fid = h5open(filename, fmode) > # matrix to write in chunks > B = rand(64,64,1000) > # figure out its dimensions > sizeTuple = size(B) > Ndims = length(sizeTuple) > # set up to write in chunks of sizeArray > sizeArray = ones(Int, Ndims) > [sizeArray[i] = sizeTuple[i] for i in 1:(Ndims-1)] # last value of size > array is :...:,1 > # create a dataset models within root > dset = d_create(fid, "models", datatype(Float64), dataspace(size(B)), > "chunk", sizeArray) > [dset[:,:,i] = slicedim(B, Ndims, i) for i in 1:size(B, Ndims)] > close(fid) > > This works, but the second last line, dset[:,:,i] requires syntax > specific to writing a slice of a dimension 3 array - but I don't know the > dimensions until run time. Of course I could just write to a flat binary > file incrementally, but HDF5.jl could make my life so much simpler! > > Many thanks for any pointers. >
