Ramiro
I think the problem is the loop - R doesn't release memory allocated inside
an expression until the expression completes. A for loop is an expression,
so it duplicates fit and dataset on every iteration. An alternative
approach that I have found successful in similar circumstances is to use
sapply(), like this
fits <- list()
sapply(1:N,function(i){
dataset <- generateDataset(i)
fit[[i]] <- try( memoryHogFunction(dataset, otherParameters))
})
I'm assuming above that you want to save the result of memoryHogFunction
from each iteration.
hth
Drew
On Thu, Apr 5, 2012 at 8:35 AM, Ramiro Barrantes <
ram...@precisionbioassay.com> wrote:
> Dear list,
>
> I am trying to reclaim what I think is lost memory in R, I have been using
> gc(), rm() and also using Rprof to figure out where all the memory is going
> but I might be missing something.
>
> I have the following situation
>
> basic loop which calls memoryHogFunction:
>
> for i in (1:N) {
> dataset <- generateDataset(i)
> fit <- try( memoryHogFunction(dataset, otherParameters))
> }
>
> and within
>
> memoryHogFunction <- function(dataset, params){
>
> fit <- try(nlme(someinitialValues)
> ...
> fit <- try(updatenlme(otherInitialValues)
> ...
> fit <- try(updatenlme(otherInitialValues)
> ...
> ret <- fit ( and other things)
> return a result "ret"
> }
>
> The problem is that, memoryHogFunction uses a lot of memory, and at the
> end returns a result (which is not big) but the memory used by the
> computation seems to be still occupied. The original loop continues, but
> the memory used by the program grows and grows after each call to
> memoryHogFunction.
>
> I have been trying to do gc() after each run in the loop, and have even
> done:
>
> in memoryHogFunction()
> ...
> ret <- fit ( and other things)
> rm(list=ls()[-match("ret",ls())])
> return a result "ret"
> }
>
> ???
>
> A typical results from gc() after each loop iteration says:
> used (Mb) gc trigger (Mb) max used (Mb)
> Ncells 326953 17.5 597831 32.0 597831 32.0
> Vcells 1645892 12.6 3048985 23.3 3048985 23.3
>
> Which doesn't reflect that 340mb (and 400+mb in virtual memory) that are
> being used right now.
>
> Even when I do:
>
> print(sapply(ls(all.names=TRUE), function(x) object.size(get(x))))
>
> the largest object is 8179808, which is what it should be.
>
> THe only thing that looked suspicious was the following within Rprof (with
> memory=stats option), the tot.duplications might be a problem???:
>
> index: "with":"with.default"
> vsize.small max.vsize.small vsize.large max.vsize.large
> 30841 63378 20642 660787
> nodes max.nodes duplications tot.duplications
> 3446132 8115016 12395 61431787
> samples
> 4956
>
> Any suggestions? Is it something about the use of loops in R? Is it
> maybe the try's???
>
> Thanks in advance for any help,
>
> Ramiro
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
--
Drew Tyre
School of Natural Resources
University of Nebraska-Lincoln
416 Hardin Hall, East Campus
3310 Holdrege Street
Lincoln, NE 68583-0974
phone: +1 402 472 4054
fax: +1 402 472 2946
email: aty...@unl.edu
http://snr.unl.edu/tyre
http://aminpractice.blogspot.com
http://www.flickr.com/photos/atiretoo
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
