Hello,
l want to store each tuple X[p] in a different file
# X[1] in mini_batch1.jld X[2] in mini_batch2.....
# but my code below stores (duplicate) all the tuple of X[p] in the files
created.
let's see an example :
m= 100 k= 3 # number of tuples or partition
y=rand_gen(m,k)
(3,[[-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,
1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-
1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0],[1.0,-1.0,-1.0,-1.0,
1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.
0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,
-1.0,-1.0],[1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.
0,1.0,-1.0,-1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,1.
0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0]])
l want to have in : mini_batch1 the first tuple
[-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,
1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,
1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0]
mini_batch2 the second tuple [1.0,-1.0,-1.0,-1.0,1.0,1.0,1.
0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,
-1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,-1.0]
and so on.
However my code do the job of creating the mini_batches files but fail to
store one by one tuple. how can l fix that ?
workspace()
using JLD, HDF5
function gen_random(m,k)
# m the length of the vector , for instance m=100000 and k the number of
partitions let's set k=16
s = rand(m)
# Pkg.add("JLD"), Pkg.add("HDF5") these two packages are needed in order to
store our vectors in files under the extension jld
# allow to convert each random number to -1 or 1
X=float_to_binary(s)
parts= kfoldperm(length(X),k)
# l want to store each tuple X[p] in a different file
# X[1] in mini_batch1.jld X[2] in mini_batch2.....
# but my code below store all the tuple X[p] in the files created.
for p in 1:length(parts)
file =jldopen(@sprintf("my path to file/mini_batch%d.jld", p),"w")
write(file, "X", [X[p] for p in parts])
close(file)
end
return [X[p] for p in parts]
function float_to_binary(s,level=0.4)
for i=1:length(s)
s[i] = s[i] > level ? 1.0 : -1.0
end
file = jldopen("/my path/mydata.jld", "w")
write(file, "s", s) # alternatively, say "@write file A"
close(file)
return s
end
function kfoldperm(l,k)
n,r = divrem(l,k)
b = collect(1:n:l+1)
for i in 1:length(b)
b[i] += i > r ? r : i-1
end
p = randperm(l)
return [p[r] for r in [b[i]:b[i+1]-1 for i=1:k]]
end
