Yes, it sounds like this was indeed the problem. I bet this could be solved automatically with a proper finalizer. Is there any chance you could create a small testcase that demonstrates the problem, and then submit an issue? https://github.com/JuliaLang/julia/issues/new
Best, --Tim On Wednesday, May 21, 2014 03:04:38 PM Gustavo Camilo wrote: > Hi Tim, thanks for answering! > > I don't do any explicit passing, all memory management is done through a > pmap(data, x->myFunction(x,capitals)) > > I was able to solve(?) the problem (has been running for a while, way > longer than the time it took to crash before) by calling instead > > @everywhere capitals = 0 > capitalsRaw = 0 > gc() > > That is, deallocating with an @everywhere sign. Do you think this could be > what caused the problem before? I think that before the garbage collector > let them stay in memory as we did not deallocate locally on each processor. > > I have not been able to do a simple testcase as I'm still using the > machine to compute some stuff, but will try to do so soon. > > Gustavo > > On Monday, 19 May 2014 16:53:18 UTC-4, Tim Holy wrote: > > I haven't tested this myself, but is there any chance that a reference to > > the > > SharedArray survives in one of the workers? For example, if you did a > > > > rr = put!(RemoteRef(pworker), SA) > > > > then you'd have to do the same tricks in the worker that you did in the > > driver > > process. > > > > But if you've checked the obvious candidates, you might as well file an > > issue. > > It would be helpful if you can produce a small testcase that reproduces > > the > > problem. > > > > --Tim > > > > On Monday, May 19, 2014 10:31:41 AM Gustavo Camilo wrote: > > > Hi all, > > > > > > I've run across a memory problem that I want to know if I'm doing wrong > > > before I open an issue: > > > > > > In the function I optimize I run a large procedure that calls C code > > > repeatedly and I end up having a huge (40^5 Float64) matrix, which I > > > > want > > > > > to be able to use in a function I pmap, so I run this code to compute > > > > and > > > > > then copy into a shared array: > > > > > > > > > capitalsRaw = findZeroDividendsInC(privModel) > > > capitals = > > > > SharedArray(Float64,(length(privModel.vW),3,length(privModel.vK),length(pr > > iv> > > > Model.vQ), > > > length(privModel.vQ),length(privModel.vQ)),init=false,pids=workers()); > > > copy!(capitals,capitalsRaw) > > > > > > I do some work with this, and then, since my memory usage was > > > > increasing, I > > > > > release these arrays like it says in the Performance section of the > > > > manual: > > > capitals = 0 > > > capitalsRaw = 0 > > > gc() > > > > > > > > > However, after the function has been called a bunch of times: > > > > > > exception on > > > > > > From worker 17: System max size of single shmem segment(MB) : > > 65536 > > > > > From worker 17: System max size of all shmem segments(MB) : 16777216 > > > From worker 17: Requested size(MB) : 1221 > > > From worker 17: Please ensure requested size is within system limits. > > > From worker 17: If not, increase system limits and try again. > > > ERROR: memory mapping failed: Cannot allocate memory > > > > > > in systemerror at error.jl:39 > > > in mmap at mmap.jl:32 > > > in mmap_array at mmap.jl:107 > > > in shm_mmap_array at sharedarray.jl:357 > > > in anonymous at sharedarray.jl:59 > > > in anonymous at multi.jl:840 > > > in run_work_thunk at multi.jl:613 > > > in run_work_thunk at multi.jl:622 > > > in anonymous at task.jl:6 > > > > > > Shouldn't releasing the shared arrays by setting them to zero and > > > > calling > > > > > gc() have cleaned up my usage? I don't think there is any other leak in > > > > the > > > > > code, and the error is from the shared memory array allocator always, > > > > not > > > > > for any other part of the code. > > > > > > Thanks for any help! > > > Gustavo
