It's pretty simple. In R, objects are stored im memory. If the objects are big, they take up a lot of memory. If your data frame is big, the objects are big too.
The problem really only arises commonly if you are fitting many models simultaneously and holding them all in memory at once, but it can arise if your data frame is truly humongous, and you are unnecessarily holding multiple copies of it in several fitted model objects. Bill Venables http://www.cmis.csiro.au/bill.venables/ -----Original Message----- From: Juliet Hannah [mailto:[email protected]] Sent: Sunday, 15 February 2009 2:25 PM To: Venables, Bill (CMIS, Cleveland) Cc: [email protected]; [email protected] Subject: Re: [R] Issue with step(): Fails to look for object$model Could you explain how memory problems may arise? I understand that by using model=FALSE, we can reduce the memory required. But if we did not, what kinds of problems may arise. I have run into a possible memory leak that I have not been able to work around (using yags, not lm). I am trying to gain insight into this problem. Thanks, Juliet On Sat, Feb 14, 2009 at 10:18 PM, <[email protected]> wrote: > Without arguing you case I would point out that the data need not be there in > l$model: > >> l <- lm(y ~ x, data=ex, model = FALSE) >> l > > Call: > lm(formula = y ~ x, data = ex, model = FALSE) > > Coefficients: > (Intercept) x > 0.1310 -0.1736 > >> l$model > NULL > > Model objects are often huge simply because by default they squirrel away a > copy of the original data inside themselves. One way to avoid memory > problems can be only to keep this backup copy within the fitted model object > only when you really need to do so, which is not all that often, in fact. > > I can see why step() (and allies) do not work the way you think they should. > These functions use the call component of the fitted model object and modify > that. And if the call component says your data are in the data frame 'ex', > they take you seriously. They your word for it. We're not dealing with > Microsoft software here, you know. > > Bill Venables > http://www.cmis.csiro.au/bill.venables/ > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Adam D. I. Kramer > Sent: Sunday, 15 February 2009 12:44 PM > To: [email protected] > Subject: [R] Issue with step(): Fails to look for object$model > > Hi, > > I'm playing around with stepwise regression, using the step > function, and believe I have found a bug (or at least, a strong case for > improvement): > >> ex <- data.frame(y=rnorm(100),x=rnorm(100)) >> l <- lm(y ~ x, data=ex) >> step(l) > [output is correct] >> rm(ex) >> step(l) > Start: AIC=11.79 > y ~ x > > Df Sum of Sq RSS AIC > - x 1 0.120 108.221 9.900 > <none> 108.100 11.789 > Error in inherits(x, "data.frame") : object "ex" not found > > ...ex is not found, so step fails. However, all of the necessary data to run > the step function is present in l$model. > > I would also argue that step() *should* use l$model if at all possible, as > it seems reasonable to expect that ex may undergo changes. Further, step() > does not appear to test any other columns present in ex (even if > direction="both" is specified), unless they are specified in scope. > > If I am misunderstanding step() or if there is a good reason why it operates > this way, please let me know! > > --Adam > > ______________________________________________ > [email protected] mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > ______________________________________________ > [email protected] mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ [email protected] mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
