Hi Jim, Thanks for your prompt response, I am using a fairly powerful Mac with Leopard OS and 17GB RAM and 2x3 GhZ intel zeon processor so I do not think the system is paging. I also using the Rmpi and snow utilities to parallelize it but even then it takes 3.5-4 hours to just complete one chunk of matrices. You mentioned about storing the data and applying on 1 column at a time. Any hint on how I should I go about doing that? I cam across the filehash package but am not sure how to use apply over an environment variable. So any help in this direction will be most welcome. thanks
---- Original message ---- >Date: Tue, 29 Apr 2008 16:05:41 -0400 >From: "jim holtman" <[EMAIL PROTECTED]> >Subject: Re: [R] Applying user function over a large matrix >To: "Sudipta Sarkar" <[EMAIL PROTECTED]> > >What size machine do you have. A single copy of your object will >require 1.5GB of memory. How slow is slow? Is the operating system >paging because it does not have enough physical memory? can you store >the data and only operate on 1 column at a time -- this reduces the >size of the object to 72MB. > >On Tue, Apr 29, 2008 at 3:16 PM, Sudipta Sarkar <[EMAIL PROTECTED]> wrote: >> Respected R experts, >> I am trying to apply a user function that basically calls and >> applies the R loess function from stat package over each time >> series. I have a large matrix of size 21 X 9000000 and I need >> to apply the loess for each column and hence I have >> implemented this separate user function that applies loess >> over each column and I am calling this function foo as follows: >> xc<-apply(t,2,foo) where t is my 21 X 9000000 matrix and >> loess. This is turning out to be a very slow process and I >> need to repeat this step for 25-30 such large matrix chunks. >> Is there any trick I can use to make this work faster? >> Any help will be deeply appreciated. >> Regards >> >> >> Sudipta Sarkar PhD >> Senior Analyst/Scientist >> Lanworth Inc. (Formerly Forest One Inc.) >> 300 Park Blvd., Ste 425 >> Itasca, IL >> Ph: 630-250-0468 >> >> ______________________________________________ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > >-- >Jim Holtman >Cincinnati, OH >+1 513 646 9390 > >What is the problem you are trying to solve? Sudipta Sarkar PhD Senior Analyst/Scientist Lanworth Inc. (Formerly Forest One Inc.) 300 Park Blvd., Ste 425 Itasca, IL Ph: 630-250-0468 ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
