are the "chunks" on which you need to apply the function rolling windows? do they overlap?
I have some c++ template utilities that I use for window functions (on timeseries objects) which you are welcome to copy and modify to fit your problem. they are available here: git://repo.or.cz/fts.git git://repo.or.cz/tslib.git -Whit On Tue, Apr 29, 2008 at 4:28 PM, Sudipta Sarkar <[EMAIL PROTECTED]> wrote: > Hi Jim, > Thanks for your prompt response, > > I am using a fairly powerful Mac with Leopard OS and 17GB RAM > and 2x3 GhZ intel zeon processor so I do not think the system > is paging. I also using the Rmpi and snow utilities to > parallelize it but even then it takes 3.5-4 hours to just > complete one chunk of matrices. > You mentioned about storing the data and applying on 1 column > at a time. Any hint on how I should I go about doing that? I > cam across the filehash package but am not sure how to use > apply over an environment variable. So any help in this > direction will be most welcome. > thanks > > ---- Original message ---- > >Date: Tue, 29 Apr 2008 16:05:41 -0400 > >From: "jim holtman" <[EMAIL PROTECTED]> > >Subject: Re: [R] Applying user function over a large matrix > >To: "Sudipta Sarkar" <[EMAIL PROTECTED]> > > > >What size machine do you have. A single copy of your object will > >require 1.5GB of memory. How slow is slow? Is the operating > system > >paging because it does not have enough physical memory? can > you store > >the data and only operate on 1 column at a time -- this > reduces the > >size of the object to 72MB. > > > >On Tue, Apr 29, 2008 at 3:16 PM, Sudipta Sarkar > <[EMAIL PROTECTED]> wrote: > >> Respected R experts, > >> I am trying to apply a user function that basically calls and > >> applies the R loess function from stat package over each time > >> series. I have a large matrix of size 21 X 9000000 and I need > >> to apply the loess for each column and hence I have > >> implemented this separate user function that applies loess > >> over each column and I am calling this function foo as follows: > >> xc<-apply(t,2,foo) where t is my 21 X 9000000 matrix and > >> loess. This is turning out to be a very slow process and I > >> need to repeat this step for 25-30 such large matrix chunks. > >> Is there any trick I can use to make this work faster? > >> Any help will be deeply appreciated. > >> Regards > >> > >> > >> Sudipta Sarkar PhD > >> Senior Analyst/Scientist > >> Lanworth Inc. (Formerly Forest One Inc.) > >> 300 Park Blvd., Ste 425 > >> Itasca, IL > >> Ph: 630-250-0468 > >> > >> ______________________________________________ > >> R-help@r-project.org mailing list > >> https://stat.ethz.ch/mailman/listinfo/r-help > >> PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > >> and provide commented, minimal, self-contained, > reproducible code. > >> > > > > > > > >-- > >Jim Holtman > >Cincinnati, OH > >+1 513 646 9390 > > > >What is the problem you are trying to solve? > > > Sudipta Sarkar PhD > Senior Analyst/Scientist > Lanworth Inc. (Formerly Forest One Inc.) > 300 Park Blvd., Ste 425 > Itasca, IL > Ph: 630-250-0468 > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
