Duplicated in
http://stackoverflow.com/questions/38147946/julia-how-to-store-tuples-in-different-files
--Tim
On Friday, July 1, 2016 7:48:34 AM CDT Ahmed Mazari wrote:
> Hello,
>
> l want to store each tuple X[p] in a different file
> # X[1] in mini_batch1.jld X[2] in mini_batch2.....
> # but my code below stores (duplicate) all the tuple of X[p] in the files
> created.
>
>
>
> let's see an example :
>
> m= 100 k= 3 # number of tuples or partition
> y=rand_gen(m,k)
>
> (3,[[-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,
> 1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-
> 1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0],[1.0,-1.0,-1.0,-1.0,
> 1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.
> 0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,
> -1.0,-1.0],[1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.
> 0,1.0,-1.0,-1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,1.
> 0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0]])
>
> l want to have in : mini_batch1 the first tuple
> [-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,
> 1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,
> 1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0]
>
>
> mini_batch2 the second tuple [1.0,-1.0,-1.0,-1.0,1.0,1.0,1.
> 0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,
> -1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,-1.0]
>
> and so on.
>
> However my code do the job of creating the mini_batches files but fail to
> store one by one tuple. how can l fix that ?
> workspace()
> using JLD, HDF5
> function gen_random(m,k)
>
> # m the length of the vector , for instance m=100000 and k the number of
> partitions let's set k=16
>
> s = rand(m)
> # Pkg.add("JLD"), Pkg.add("HDF5") these two packages are needed in order to
> store our vectors in files under the extension jld
>
> # allow to convert each random number to -1 or 1
>
> X=float_to_binary(s)
> parts= kfoldperm(length(X),k)
> # l want to store each tuple X[p] in a different file
> # X[1] in mini_batch1.jld X[2] in mini_batch2.....
> # but my code below store all the tuple X[p] in the files created.
> for p in 1:length(parts)
> file =jldopen(@sprintf("my path to file/mini_batch%d.jld", p),"w")
> write(file, "X", [X[p] for p in parts])
> close(file)
> end
> return [X[p] for p in parts]
>
> function float_to_binary(s,level=0.4)
> for i=1:length(s)
> s[i] = s[i] > level ? 1.0 : -1.0
> end
> file = jldopen("/my path/mydata.jld", "w")
> write(file, "s", s) # alternatively, say "@write file A"
> close(file)
> return s
> end
>
>
> function kfoldperm(l,k)
> n,r = divrem(l,k)
> b = collect(1:n:l+1)
> for i in 1:length(b)
> b[i] += i > r ? r : i-1
> end
> p = randperm(l)
> return [p[r] for r in [b[i]:b[i+1]-1 for i=1:k]]
>
>
> end
