Hi Tim, thanks for answering! I don't do any explicit passing, all memory management is done through a pmap(data, x->myFunction(x,capitals))
I was able to solve(?) the problem (has been running for a while, way longer than the time it took to crash before) by calling instead @everywhere capitals = 0 capitalsRaw = 0 gc() That is, deallocating with an @everywhere sign. Do you think this could be what caused the problem before? I think that before the garbage collector let them stay in memory as we did not deallocate locally on each processor. I have not been able to do a simple testcase as I'm still using the machine to compute some stuff, but will try to do so soon. Gustavo On Monday, 19 May 2014 16:53:18 UTC-4, Tim Holy wrote: > > I haven't tested this myself, but is there any chance that a reference to > the > SharedArray survives in one of the workers? For example, if you did a > rr = put!(RemoteRef(pworker), SA) > then you'd have to do the same tricks in the worker that you did in the > driver > process. > > But if you've checked the obvious candidates, you might as well file an > issue. > It would be helpful if you can produce a small testcase that reproduces > the > problem. > > --Tim > > On Monday, May 19, 2014 10:31:41 AM Gustavo Camilo wrote: > > Hi all, > > > > I've run across a memory problem that I want to know if I'm doing wrong > > before I open an issue: > > > > In the function I optimize I run a large procedure that calls C code > > repeatedly and I end up having a huge (40^5 Float64) matrix, which I > want > > to be able to use in a function I pmap, so I run this code to compute > and > > then copy into a shared array: > > > > > > capitalsRaw = findZeroDividendsInC(privModel) > > capitals = > > > SharedArray(Float64,(length(privModel.vW),3,length(privModel.vK),length(priv > > > Model.vQ), > > length(privModel.vQ),length(privModel.vQ)),init=false,pids=workers()); > > copy!(capitals,capitalsRaw) > > > > I do some work with this, and then, since my memory usage was > increasing, I > > release these arrays like it says in the Performance section of the > manual: > > > > > > capitals = 0 > > capitalsRaw = 0 > > gc() > > > > > > However, after the function has been called a bunch of times: > > > > exception on > > From worker 17: System max size of single shmem segment(MB) : > 65536 > > From worker 17: System max size of all shmem segments(MB) : 16777216 > > From worker 17: Requested size(MB) : 1221 > > From worker 17: Please ensure requested size is within system limits. > > From worker 17: If not, increase system limits and try again. > > ERROR: memory mapping failed: Cannot allocate memory > > in systemerror at error.jl:39 > > in mmap at mmap.jl:32 > > in mmap_array at mmap.jl:107 > > in shm_mmap_array at sharedarray.jl:357 > > in anonymous at sharedarray.jl:59 > > in anonymous at multi.jl:840 > > in run_work_thunk at multi.jl:613 > > in run_work_thunk at multi.jl:622 > > in anonymous at task.jl:6 > > > > > > Shouldn't releasing the shared arrays by setting them to zero and > calling > > gc() have cleaned up my usage? I don't think there is any other leak in > the > > code, and the error is from the shared memory array allocator always, > not > > for any other part of the code. > > > > Thanks for any help! > > Gustavo >
